Memory & CPU

This page describes the memory & cpu prerequisites for Lenses.

This documentation provides memory recommendations for Lenses.io, considering the number of Kafka topics, the number of schemas, and the complexity of these schemas (measured by the number of fields). Proper memory allocation ensures optimal performance and stability of Lenses.io in various environments.

Key Considerations

• Number of Topics: Kafka topics require memory for indexing, metadata, and state management.

• Schemas and Their Complexity: The memory impact of schemas is influenced by both the number of schemas and the number of fields within each schema. Each schema field contributes to the creation of Lucene indexes, which affects memory usage.

Baseline Memory Requirements

For a basic setup with minimal topics and schemas:

• Minimum Memory: 4 GB

• Recommended Memory: 8 GB

This setup assumes:

• Fewer than 100 topics

• Fewer than 100 schemas

• Small schemas with few fields (less than 10 fields per schema)

Scaling with Topics

Memory requirements increase with the number of topics. Topics are used as the primary reference for memory scaling, with additional considerations for schemas.

Number of Topics / Partitions	Recommended Memory
Up to 1,000 / 10,000 partitions	12 GB
1,001 to 10,000 / 100.000 partitions	24 GB
10,001 to 30,000 / 300.000 partitions	64 GB

Impact of Schemas and Their Complexity

Schemas have a significant impact on memory usage, particularly as the number of fields within each schema increases. The memory impact is determined by both the number of schemas and the complexity (number of fields) of these schemas.

Memory Addition Based on Schema Complexity

Schema Complexity	Number of Fields per Schema	Memory Addition
Low to Moderate Complexity	Up to 50 fields	None
High Complexity	51 - 100 fields	1 GB for every 1,000 schemas
Very High Complexity	100+ fields	2 GB for every 1,000 schemas

Cross-Reference Table for Topics and Schema Complexity

Number of Topics	Number of Schemas	Number of Fields per Schema	Base Memory	Additional Memory	Total Recommended Memory
1,000	1,000	Up to 10	8 GB	None	12 GB
1,000	1,000	11 - 50	8 GB	None	12 GB
5,000	5,000	Up to 10	12 GB	None	16 GB
5,000	5,000	11 - 50	12 GB	None	16 GB
10,000	10,000	Up to 10	16 GB	None	24 GB
10,000	10,000	51 - 100	24 GB	10 GB	34 GB
30,000	30,000	Up to 10	64 GB	None	64 GB
30,000	30,000	51 - 100	64 GB	30 GB	94 GB

Example Configurations

To help illustrate how to apply these recommendations, here are some example configurations considering both topics and schema complexity:

Small Setup

• Topics: 500

• Schemas: 100 (average size 50 KB, 8 fields per schema)

Recommended Memory: 8 GB

• Schema Complexity: Low → No additional memory needed.

Total Recommended Memory: 8 GB

Medium Setup

• Topics: 5,000

• Schemas: 1,000 (average size 200 KB, 25 fields per schema)

Base Memory: 12 GB

• Schema Complexity: Moderate → No additional memory needed.

Total Recommended Memory: 16 GB

Large Setup

• Topics: 15,000

• Schemas: 3,000 (average size 500 KB, 70 fields per schema)

Base Memory: 32 GB

• Schema Complexity: High → Add 3 GB for schema complexity.

Total Recommended Memory: 35 GB

High-Volume Setup Examples

• 30,000 Topics

  • Schemas: 5,000 (average size 300 KB, 30 fields per schema)

  Base Memory: 64 GB

  • Schema Complexity: Moderate → Add 5 GB for schema complexity.

  Total Recommended Memory: 69 GB

Additional Considerations

• High Throughput: If your Kafka cluster is expected to handle high throughput, consider adding 20-30% more memory than the recommendations.

• Complex Queries and Joins: If using Lenses.io for complex data queries and joins, consider increasing the memory allocation by 10-15% to accommodate the additional processing.

• Monitoring and Adjustment: Regularly monitor memory usage and adjust based on actual load and performance.

Conclusion

Proper memory allocation is crucial for the performance and reliability of Lenses.io, especially in environments with a large number of topics and complex schemas. While topics provide a solid baseline for memory recommendations, the complexity of schemas—particularly the number of fields—can also significantly impact memory usage. Regular monitoring and adjustments are recommended to ensure that your Lenses.io setup remains performant as your Kafka environment scales.