Kafka supports Quotas on requests to control the resources of your cluster. Quotas is a useful mechanism to operate clusters with multiple tenants or when your producers/consumers
use high volumes of data. In any case, you want to protect your cluster from applications that may monopolize your resources.
How quotas work?
According to Kafka documentation:
By default, each unique client group receives a fixed quota as configured by the cluster. This quota is defined on a per-broker basis. Each client can utilize this quota per broker before it gets throttled.
Use quotas to:
Access Management & permissions
To manage quotas on Kafka, you need to:
Cloud and managed Kafka services do not currently offer access to Zookeeper and quota management will become available via KIP-546._
You can create new Quotas, with the New Quota button and selecting the client or user (or combination) to apply this quota on. You can also set up default quotas.
For example, to apply a Quota on a particular application that can create an aggressive spike on your data platform (i.e. an Apache Spark application), select CLIENT:
And for example, enforce that specific application to not produce and consumer more than 5MBytes/sec:
With quota management enabled, you can create, search, and manage your Quotas:
You can enforce quotas on produce and fetch requests for:
The following order of precedence is followed when multiple quotas are in place.
Learn more about Kafka Quotas.
Kafka Quotas are also supported by the CLI to enable automation scenarios.
CLI - API
On this page