# Memory & CPU

This documentation provides memory recommendations for Lenses.io, considering the number of Kafka topics, the number of schemas, and the complexity of these schemas (measured by the number of fields). Proper memory allocation ensures optimal performance and stability of Lenses.io in various environments.

## Key Considerations

* **Number of Topics**: Kafka topics require memory for indexing, metadata, and state management.
* **Schemas and Their Complexity**: The memory impact of schemas is influenced by both the number of schemas and the number of fields within each schema. Each schema field contributes to the creation of Lucene indexes, which affects memory usage.

## Baseline Memory Requirements

For a basic setup with minimal topics and schemas:

* **Minimum Memory**: 4 GB
* **Recommended Memory**: 8 GB

This setup assumes:

* Fewer than 100 topics
* Fewer than 100 schemas
* Small schemas with few fields (less than 10 fields per schema)

## Scaling with Topics

Memory requirements increase with the number of topics. Topics are used as the primary reference for memory scaling, with additional considerations for schemas.

| **Number of Topics / Partitions**         | **Recommended Memory** |
| ----------------------------------------- | ---------------------- |
| **Up to 1,000 / 10,000 partitions**       | 12 GB                  |
| **1,001 to 10,000 / 100.000 partitions**  | 24 GB                  |
| **10,001 to 30,000 / 300.000 partitions** | 64 GB                  |

## Impact of Schemas and Their Complexity

Schemas have a significant impact on memory usage, particularly as the number of fields within each schema increases. The memory impact is determined by both the number of schemas and the complexity (number of fields) of these schemas.

#### Memory Addition Based on Schema Complexity

| **Schema Complexity**          | **Number of Fields per Schema** | **Memory Addition**          |
| ------------------------------ | ------------------------------- | ---------------------------- |
| **Low to Moderate Complexity** | Up to 50 fields                 | None                         |
| **High Complexity**            | 51 - 100 fields                 | 1 GB for every 1,000 schemas |
| **Very High Complexity**       | 100+ fields                     | 2 GB for every 1,000 schemas |

## Cross-Reference Table for Topics and Schema Complexity

<table data-header-hidden><thead><tr><th></th><th></th><th></th><th width="103"></th><th width="112"></th><th></th></tr></thead><tbody><tr><td><strong>Number of Topics</strong></td><td><strong>Number of Schemas</strong></td><td><strong>Number of Fields per Schema</strong></td><td><strong>Base Memory</strong></td><td><strong>Additional Memory</strong></td><td><strong>Total Recommended Memory</strong></td></tr><tr><td>1,000</td><td>1,000</td><td>Up to 10</td><td>8 GB</td><td>None</td><td>12 GB</td></tr><tr><td>1,000</td><td>1,000</td><td>11 - 50</td><td>8 GB</td><td>None</td><td>12 GB</td></tr><tr><td>5,000</td><td>5,000</td><td>Up to 10</td><td>12 GB</td><td>None</td><td>16 GB</td></tr><tr><td>5,000</td><td>5,000</td><td>11 - 50</td><td>12 GB</td><td>None</td><td>16 GB</td></tr><tr><td>10,000</td><td>10,000</td><td>Up to 10</td><td>16 GB</td><td>None</td><td>24 GB</td></tr><tr><td>10,000</td><td>10,000</td><td>51 - 100</td><td>24 GB</td><td>10 GB</td><td>34 GB</td></tr><tr><td>30,000</td><td>30,000</td><td>Up to 10</td><td>64 GB</td><td>None</td><td>64 GB</td></tr><tr><td>30,000</td><td>30,000</td><td>51 - 100</td><td>64 GB</td><td>30 GB</td><td>94 GB</td></tr></tbody></table>

## Example Configurations

To help illustrate how to apply these recommendations, here are some example configurations considering both topics and schema complexity:

### Small Setup

* **Topics**: 500
* **Schemas**: 100 (average size 50 KB, 8 fields per schema)

**Recommended Memory**: 8 GB

* **Schema Complexity**: Low → No additional memory needed.

**Total Recommended Memory**: 8 GB

### Medium Setup

* **Topics**: 5,000
* **Schemas**: 1,000 (average size 200 KB, 25 fields per schema)

**Base Memory**: 12 GB

* **Schema Complexity**: Moderate → No additional memory needed.

**Total Recommended Memory**: 16 GB

### Large Setup

* **Topics**: 15,000
* **Schemas**: 3,000 (average size 500 KB, 70 fields per schema)

**Base Memory**: 32 GB

* **Schema Complexity**: High → Add 3 GB for schema complexity.

**Total Recommended Memory:** 35 GB

### High-Volume Setup Examples

* **30,000 Topics**

  * **Schemas**: 5,000 (average size 300 KB, 30 fields per schema)

  **Base Memory**: 64 GB

  * **Schema Complexity**: Moderate → Add 5 GB for schema complexity.

  **Total Recommended Memory**: 69 GB

## Additional Considerations

* **High Throughput**: If your Kafka cluster is expected to handle high throughput, consider adding 20-30% more memory than the recommendations.
* **Complex Queries and Joins**: If using Lenses.io for complex data queries and joins, consider increasing the memory allocation by 10-15% to accommodate the additional processing.
* **Monitoring and Adjustment**: Regularly monitor memory usage and adjust based on actual load and performance.

### Conclusion

Proper memory allocation is crucial for the performance and reliability of Lenses.io, especially in environments with a large number of topics and complex schemas. While topics provide a solid baseline for memory recommendations, the complexity of schemas—particularly the number of fields—can also significantly impact memory usage. Regular monitoring and adjustments are recommended to ensure that your Lenses.io setup remains performant as your Kafka environment scales.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.lenses.io/latest/deployment/configuration/agent/memory-and-cpu.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.