4.0

You are viewing documentation for an older version of Lenses.io View latest documentation here

Schema Registry

Working with records serialized in AVRO format requires a Schema Registry. Lenses supports Confluent’s Schema Registry and any other registry that implements its API.

The most simple configuration includes the Schema Registry hosts, and whether schema deletion should be allowed.

lenses.schema.registry.urls = [
  { url: "http://host1:8081" },
  { url: "http://host2:8081" }
]

# Enable schema deletion - default : false
lenses.schema.registry.delete = true
# Kafka topic that stores the state of the Schema Registry
lenses.schema.registry.topics = "_schemas"

The scheme (http:// or https://) should always be included in the registry URLs.

More advanced setups, that include authentication and access to JMX metrics are possible.

Basic Authentication requires a user and password.

lenses.schema.registry.auth = "USER_INFO"
lenses.schema.registry.username = "USERNAME"
lenses.schema.registry.password = "PASSWORD"

lenses.kafka.settings.client.basic.auth.credentials.source = USER_INFO
lenses.kafka.settings.client.basic.auth.user.info = "USERNAME:PASSWORD"

TLS support for Schema Registry is limited in Lenses 4.0 and mostly restricted to the application —SQL Processors do not support it. Please upgrade to Lenses 4.1 or later to get full support.

When the Registry is served over TLS (encryption-in-transit) a truststore is needed if the Registry’s certificate is not signed by a trusted CA. This setting should be applied to the global truststore and Kafka client settings.

The following environment variable should be exported to set the global truststore.

LENSES_OPTS="-Djavax.net.ssl.trustStore=/path/to/truststore.jks -Djavax.net.ssl.trustStorePassword=[PASSWORD]"

The configuration for the Kafka client happens within lenses.conf.

lenses.kafka.settings.client.schema.registry.ssl.truststore.location = "/path/to/truststore.jks"
lenses.kafka.settings.client.schema.registry.ssl.truststore.password = "[PASSWORD]"

Support for TLS Authentication is even more limited, restricted to record serialization within Lenses (like the Explore and SQL Studio screens), but absent from the Schema Registry screen.

The configuration for the Kafka client happens within lenses.conf.

lenses.kafka.settings.client.schema.registry.ssl.keystore.location = "/path/to/keystore.jks"
lenses.kafka.settings.client.schema.registry.ssl.keystore.password = "[PASSWORD]"
lenses.kafka.settings.client.schema.registry.ssl.key.password      = "[KEY PASSWORD]"

Lenses can read Schema Registry metrics via JMX or the Jolokia JavaAgent. A subset of the available metrics is shown in the Services screen.

The configuration is part of the registry URLs. Any settings marked as optional can be ommited.

lenses.schema.registry.urls = [
  {
    url: "http://SR_HOST_1:8081",
    metrics: {       # Optional section
       ssl: false,   # Optional (default: false); if used - please make sure the remote 
                     # certificate is accepted by Lenses truststore
       user: "",     # Optional, the remote HTTP user
       password: "", # Optional, the remote HTTP password
       type: "JMX", 
       url: "SR_HOST_1:9583" # no scheme should be included, just host and port
    }
  },
  {
    url: "http://SR_HOST_2:8081",
     metrics: {      # Optional section
       ssl: false,   # Optional (default: false); if used - please make sure the remote 
                     # certificate is accepted by Lenses truststore
       user: "",     # Optional, the remote HTTP user
       password: "", # Optional, the remote HTTP password
       type: "JMX",  
       url: "SR_HOST_2:9583" # no scheme should be included, just host and port
     }
  }
]

Lenses can read Schema Registry metrics via JMX or the Jolokia JavaAgent. A subset of the available metrics is shown in the Services screen.

The configuration is part of the registry URLs. Any settings marked as optional can be ommited.

lenses.schema.registry.urls = [
  {
    url: "http://SR_HOST_1:8081",
    metrics: {       # Optional section
       ssl: false,   # Optional (default: false); if used - please make sure the remote 
                     # certificate is accepted by Lenses truststore
       user: "",     # Optional, the remote HTTP user
       password: "", # Optional, the remote HTTP password
       type: "JOLOKIAP",     # One of 'JOLOKIAG' (for GET requests), 'JOLOKIAP' (for POST requests)
       url: "http://host1:9585/jolokia/" # notice the scheme and trailing slash - they are both required for standard Jolokia configuration
    }
  },
  {
    url: "http://SR_HOST_2:8081",
     metrics: {      # Optional section
       ssl: false,   # Optional (default: false); if used - please make sure the remote 
                     # certificate is accepted by Lenses truststore
       user: "",     # Optional, the remote HTTP user
       password: "", # Optional, the remote HTTP password
       type: "JOLOKIAP",     # One of 'JOLOKIAG' (for GET requests), 'JOLOKIAP' (for POST requests)
       url: "http://host2:9585/jolokia/" # notice the scheme and trailing slash - they are both required for standard Jolokia configuration
     }
  }
]

Aiven offers a Schema Registry as an optional service on top of the Kafka service. It uses Basic HTTP Authentication.

Get the address and credentials from Aiven Console.

lenses.schema.registry.urls = [{url:"https://[CLUSTER-NAME]-[PROJECT-NAME].aivencloud.com:[PORT]"}]

lenses.schema.registry.auth = "USER_INFO"
lenses.schema.registry.username = "[USERNAME]"
lenses.schema.registry.password = "[PASSWORD]"

lenses.kafka.settings.client.basic.auth.credentials.source = "USER_INFO"
lenses.kafka.settings.client.basic.auth.user.info = "[USERNAME]:[PASSWORD]"

Aiven’s TLS is signed by a known CA, and does not require extra settings.


See configuration settings.