Memory & CPU

This documentation provides memory recommendations for Lenses.io, considering the number of Kafka topics, the number of schemas, and the complexity of these schemas (measured by the number of fields). Proper memory allocation ensures optimal performance and stability of Lenses.io in various environments.

Key Considerations

• Number of Topics: Kafka topics require memory for indexing, metadata, and state management.

• Schemas and Their Complexity: The memory impact of schemas is influenced by both the number of schemas and the number of fields within each schema. Each schema field contributes to the creation of Lucene indexes, which affects memory usage.

Baseline Memory Requirements

For a basic setup with minimal topics and schemas:

• Minimum Memory: 4 GB

• Recommended Memory: 8 GB

This setup assumes:

• Fewer than 100 topics

• Fewer than 100 schemas

• Small schemas with few fields (less than 10 fields per schema)

Scaling with Topics

Memory requirements increase with the number of topics. Topics are used as the primary reference for memory scaling, with additional considerations for schemas.

Impact of Schemas and Their Complexity

Schemas have a significant impact on memory usage, particularly as the number of fields within each schema increases. The memory impact is determined by both the number of schemas and the complexity (number of fields) of these schemas.

Memory Addition Based on Schema Complexity

Cross-Reference Table for Topics and Schema Complexity

Example Configurations

To help illustrate how to apply these recommendations, here are some example configurations considering both topics and schema complexity:

Small Setup

• Topics: 500

• Schemas: 100 (average size 50 KB, 8 fields per schema)

Recommended Memory: 8 GB

• Schema Complexity: Low → No additional memory needed.

Total Recommended Memory: 8 GB

Medium Setup

• Topics: 5,000

• Schemas: 1,000 (average size 200 KB, 25 fields per schema)

Base Memory: 12 GB

• Schema Complexity: Moderate → No additional memory needed.

Total Recommended Memory: 16 GB

Large Setup

• Topics: 15,000

• Schemas: 3,000 (average size 500 KB, 70 fields per schema)

Base Memory: 32 GB

• Schema Complexity: High → Add 3 GB for schema complexity.

Total Recommended Memory: 35 GB

High-Volume Setup Examples

• 30,000 Topics

  • Schemas: 5,000 (average size 300 KB, 30 fields per schema)

  Base Memory: 64 GB

  • Schema Complexity: Moderate → Add 5 GB for schema complexity.

  Total Recommended Memory: 69 GB

Additional Considerations

• High Throughput: If your Kafka cluster is expected to handle high throughput, consider adding 20-30% more memory than the recommendations.

• Complex Queries and Joins: If using Lenses.io for complex data queries and joins, consider increasing the memory allocation by 10-15% to accommodate the additional processing.

• Monitoring and Adjustment: Regularly monitor memory usage and adjust based on actual load and performance.

Conclusion

Proper memory allocation is crucial for the performance and reliability of Lenses.io, especially in environments with a large number of topics and complex schemas. While topics provide a solid baseline for memory recommendations, the complexity of schemas—particularly the number of fields—can also significantly impact memory usage. Regular monitoring and adjustments are recommended to ensure that your Lenses.io setup remains performant as your Kafka environment scales.