Kafka quotas

Introduction 

Kafka supports Quotas on requests to control the resources of your cluster. Quotas is a useful mechanism to operate clusters with multiple tenants or when your producers/consumers use high volumes of data. In any case, you want to protect your cluster from applications that may monopolize your resources.

How quotas work?

According to Kafka documentation:

By default, each unique client group receives a fixed quota as configured by the cluster. This quota is defined on a per-broker basis. Each client can utilize this quota per broker before it gets throttled.

Use quotas to:

  • deliver better SLAs
  • reduce latency
  • offer an improved multi-tenant experience and
  • provide a charge back model for cost-center allocation of the operation cost of you data platform, by applying quotas on consume/produce resources.

Required permission 

PermissionTypeDescription
Kafka Settings / ViewAdminTo view Kafka Quotas
Kafka Settings / ManageAdminTo create or delete a Kafka Quotas

Access Management & permissions

Manage Kafka quotas 

  1. From the Header bar menu, go to Admin panel.
  2. On the side navigation, select Kafka Quotas under the Kafka Settings section.
Kafka Quotas

Prerequisites 

To manage quotas on Kafka, you need to:

  1. have the Kafka Admin permission on your account
  2. have Zookeeper access. See how to configure ZK.

Cloud and managed Kafka services do not currently offer access to Zookeeper and quota management will become available via KIP-546._

Create Kafka quotas 

You can create new Quotas, with the New Quota button and selecting the client or user (or combination) to apply this quota on. You can also set up default quotas.

For example, to apply a Quota on a particular application that can create an aggressive spike on your data platform (i.e. an Apache Spark application), select CLIENT:

Add a new Kafka Quota

And for example, enforce that specific application to not produce and consumer more than 5MBytes/sec:

Create new Kafka Quota

Explore quotas 

With quota management enabled, you can create, search, and manage your Quotas:

List Kafka Quotas - Lenses.io

Quota precedence 

You can enforce quotas on produce and fetch requests for:

  • A user ( requires an authenticated Kafka )
  • An application ( using the client-id )
  • The application of a specific user ( requires an authenticated Kafka )

The following order of precedence is followed when multiple quotas are in place.

  1. (user) / (client-id)
  2. (user) / (default)
  3. (user)
  4. default / (client-id)
  5. default / clients / (default)
  6. users / default
  7. clients / (client-id)
  8. clients / default

Learn more about Kafka Quotas.

API clients 

Kafka Quotas are also supported by the CLI to enable automation scenarios.

CLI - API