This page describe the Lenses Agent configuration.
HQ's configuration is defined in the config.yaml file
To accept the Lenses EULA, set the following in the lenses.conf file:
Without accepting the EULA the Agent will not start! See License.
It has the following top level groups:
Configures authentication and authorisation.
It has the following fields:
administrators
No
[]
strings
Grants root access to principals.
users
no
[]
Array
Creates initial users for password based authentication.
Lists the names of the principals (users, service accounts) that have root access. Access control allows any API operation performed by such principals. Optional. If not set, it will default to []
.
Contains SAML2 IdP configuration. Please refer here for its structure.
Configures everything involving the HTTP.
It has the following fields:
address
Yes
n/a
string
Sets the address the HTTP server listens at.
accessControlAllowOrigin
No
["*"]
strings
Sets the value of the "Access-Control-Allow-Origin" header.
accessControlAllowCredentials
No
false
boolean
Sets the value of the "Access-Control-Allow-Credentials" header.
secureSessionCookies
No
true
boolean
Sets the "Secure" attribute on session cookies.
Sets the address the HTTP server listens at.
Example value: 127.0.0.1:80
.
Sets the value of the "Access-Control-Allow-Origin" header. This is only relevant when serving the backend from a different origin than the UI. Optional. If not set, it will default to ["*"]
.
Sets the value of the "Access-Control-Allow-Credentials" header. This is only relevant when serving the backend from a different origin than the UI. Optional. If not set, it will default to false
.
Sets the "Secure" attribute on authentication session cookies. When set, a browser sends such cookies not over unsecured HTTP (expect for localhost). If running Lenses HQ over unsecured HTTP, set this to false. Optional. If not set, it will default to true
.
Contains TLS configuration. Please refer here for its structure.
Contains SAML2 IdP configuration.
It has the following fields:
metadata
Yes
n/a
string
Contains the IdP issued XML metadata blob.
baseURL
Yes
n/a
string
Defines base URL of HQ for IdP redirects.
uiRootURL
No
/
string
Controls where to redirect to upon successful authentication.
entityID
Yes
n/a
string
Defines the Entity ID.
groupAttributeKey
No
groups
string
Sets the attribute name for group names.
userCreationMode
No
manual
string
Controls how the creation of users should be handled in relation to SSO information.
groupMembershipMode
No
manual
string
Controls how the management of a user's group membership should be handled in relation to SSO information.
Contains the IdP issued XML metadata blob.
Example value: <?xml version="1.0" ... (big blob of xml) </md:EntityDescriptor>
.
Defines the base URL of Lenses HQ; the IdP redirects back to here on success.
Example value: https://hq.example.com
.
Controls where the backend redirects to after having received a valid SAML2 assertion. Optional. If not set, it will default to /
.
Example value: /
.
Defines the Entity ID.
Example value: https://hq.example.com
.
Sets the attribute name from which group names are extracted in the SAML2 assertions. Different providers use different names. Okta, Keycloak and Google use "groups". OneLogin uses "roles". Azure uses "http://schemas.microsoft.com/ws/2008/06/identity/claims/groups". Optional. If not set, it will default to groups
.
Example value: groups
.
Controls how the creation of users should be handled in relation to SSO information. With the 'manual' mode, only users that currently exist in HQ can login. Users that do not exist are rejected. With the 'sso' mode, users that do not exist are automatically created. Allowed values are manual
or sso
. Optional. If not set, it will default to manual
.
Controls how the management of a user's group membership should be handled in relation to SSO information. With the 'manual' mode, the information about the group membership returned from an Identity Provider will not be used and a user will only be a member of groups that were explicitly assigned to him locally. With the 'sso' mode, group information from Identity Provider (IdP) will be used. On login, a user's group membership is set to the groups listed in the IdP. Groups that do not exist in HQ are ignored. Allowed values are manual
or sso
. Optional. If not set, it will default to manual
.
Controls the agent handling.
It has the following fields:
address
Yes
n/a
string
Sets the address the agent server listens at.
Sets the address the agent server listens at.
Example value: 127.0.0.1:3000
.
Contains TLS configuration. Please refer here for its structure.
Contains TLS configuration.
It has the following fields:
enabled
Yes
n/a
boolean
Enables or disables TLS.
cert
No
``
string
Sets the PEM formatted public certificate.
key
No
``
string
Sets the PEM formatted private key.
verboseLogs
No
false
boolean
Enables verbose TLS logging.
Enables or disables TLS.
Example value: false
.
Sets the PEM formatted public certificate. Optional. If not set, it will default to ``.
Example value: -----BEGIN CERTIFICATE----- EXampLeRanDoM ... -----END CERTIFICATE-----
.
Sets the PEM formatted private key. Optional. If not set, it will default to ``.
Example value: -----BEGIN PRIVATE KEY----- ExAmPlErAnDoM ... -----END PRIVATE KEY-----
.
Enables additional logging of TLS settings and events at debug level. The information presented might be a bit too much for day to day use but can provide extra information for troubleshooting TLS configuration. Optional. If not set, it will default to false
.
Configures database settings.
It has the following fields:
host
Yes
n/a
string
Sets the name of the host to connect to.
username
No
``
string
Sets the username to authenticate as.
password
No
``
string
Sets the password to authenticate as.
database
Yes
n/a
string
Sets the database to use.
schema
No
``
string
Sets the schema to use.
TLS
No
false
boolean
Enables TLS.
params
No
{}
DBConnectionParams
Provides fine-grained control.
Sets the name of the host to connect to. A comma-separated list of host names is also accepted; each host name in the list is tried in order.
Example value: postgres:5432
.
Sets the username to authenticate as. Optional. If not set, it will default to ``.
Example value: johhnybingo
.
Sets the password to authenticate as. Optional. If not set, it will default to ``.
Example value: my-password
.
Sets the database to use.
Example value: my-database
.
Sets the schema to use. Optional. If not set, it will default to ``.
Example value: my-schema
.
Enables TLS. In PostgreSQL connection string terms, setting TLS to false
corresponds to sslmode=disable
; setting TLS to true
corresponds to sslmode=verify-full
. For more fine-grained control, specify sslmode
in the params which takes precedence. Optional. If not set, it will default to false
.
Example value: true
.
Contains connection string parameters as key/values pairs. It allow fine-grained control of connection settings. The parameters can be found here: https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-PARAMKEYWORDS Optional. If not set, it will default to {}
.
Example value: {"application_name":"example"}
.
Sets the logger behaviour.
It has the following fields:
mode
Yes
n/a
string
Controls the format of the logger's output.
level
No
info
string
Controls the level of the logger.
Controls the format of the logger's output. Allowed values are text
or json
.
Controls the level of the logger. Allowed values are info
or debug
. Optional. If not set, it will default to info
.
Controls the metrics settings.
It has the following fields:
prometheusAddress
No
:9090
string
Sets the Prometheus address.
Sets the address at which Prometheus metrics are served. Optional. If not set, it will default to :9090
.
Holds the license key.
It has the following fields:
key
Yes
n/a
string
Sets the license key.
Sets the license key. An HQ key starts with "licensekey".
Accepts the Lenses EULA.
HQ's configuration is defined in the config.yaml file
To accept the Lenses EULA, set the following in the lenses.conf file:
Without accepting the EULA the Agent will not start! See License.
It has the following top level groups:
Configures authentication and authorisation.
It has the following fields:
administrators
No
[]
strings
Grants root access to principals.
users
no
[]
Array
Creates initial users for password based authentication.
Lists the names of the principals (users, service accounts) that have root access. Access control allows any API operation performed by such principals. Optional. If not set, it will default to []
.
Contains SAML2 IdP configuration. Please refer here for its structure.
Configures everything involving the HTTP.
It has the following fields:
address
Yes
n/a
string
Sets the address the HTTP server listens at.
accessControlAllowOrigin
No
["*"]
strings
Sets the value of the "Access-Control-Allow-Origin" header.
accessControlAllowCredentials
No
false
boolean
Sets the value of the "Access-Control-Allow-Credentials" header.
secureSessionCookies
No
true
boolean
Sets the "Secure" attribute on session cookies.
Sets the address the HTTP server listens at.
Example value: 127.0.0.1:80
.
Sets the value of the "Access-Control-Allow-Origin" header. This is only relevant when serving the backend from a different origin than the UI. Optional. If not set, it will default to ["*"]
.
Sets the value of the "Access-Control-Allow-Credentials" header. This is only relevant when serving the backend from a different origin than the UI. Optional. If not set, it will default to false
.
Sets the "Secure" attribute on authentication session cookies. When set, a browser sends such cookies not over unsecured HTTP (expect for localhost). If running Lenses HQ over unsecured HTTP, set this to false. Optional. If not set, it will default to true
.
Contains TLS configuration. Please refer here for its structure.
Contains SAML2 IdP configuration.
It has the following fields:
metadata
Yes
n/a
string
Contains the IdP issued XML metadata blob.
baseURL
Yes
n/a
string
Defines base URL of HQ for IdP redirects.
uiRootURL
No
/
string
Controls where to redirect to upon successful authentication.
entityID
Yes
n/a
string
Defines the Entity ID.
groupAttributeKey
No
groups
string
Sets the attribute name for group names.
userCreationMode
No
manual
string
Controls how the creation of users should be handled in relation to SSO information.
groupMembershipMode
No
manual
string
Controls how the management of a user's group membership should be handled in relation to SSO information.
Contains the IdP issued XML metadata blob.
Example value: <?xml version="1.0" ... (big blob of xml) </md:EntityDescriptor>
.
Defines the base URL of Lenses HQ; the IdP redirects back to here on success.
Example value: https://hq.example.com
.
Controls where the backend redirects to after having received a valid SAML2 assertion. Optional. If not set, it will default to /
.
Example value: /
.
Defines the Entity ID.
Example value: https://hq.example.com
.
Sets the attribute name from which group names are extracted in the SAML2 assertions. Different providers use different names. Okta, Keycloak and Google use "groups". OneLogin uses "roles". Azure uses "http://schemas.microsoft.com/ws/2008/06/identity/claims/groups". Optional. If not set, it will default to groups
.
Example value: groups
.
Controls how the creation of users should be handled in relation to SSO information. With the 'manual' mode, only users that currently exist in HQ can login. Users that do not exist are rejected. With the 'sso' mode, users that do not exist are automatically created. Allowed values are manual
or sso
. Optional. If not set, it will default to manual
.
Controls how the management of a user's group membership should be handled in relation to SSO information. With the 'manual' mode, the information about the group membership returned from an Identity Provider will not be used and a user will only be a member of groups that were explicitly assigned to him locally. With the 'sso' mode, group information from Identity Provider (IdP) will be used. On login, a user's group membership is set to the groups listed in the IdP. Groups that do not exist in HQ are ignored. Allowed values are manual
or sso
. Optional. If not set, it will default to manual
.
Controls the agent handling.
It has the following fields:
address
Yes
n/a
string
Sets the address the agent server listens at.
Sets the address the agent server listens at.
Example value: 127.0.0.1:3000
.
Contains TLS configuration. Please refer here for its structure.
Contains TLS configuration.
It has the following fields:
enabled
Yes
n/a
boolean
Enables or disables TLS.
cert
No
``
string
Sets the PEM formatted public certificate.
key
No
``
string
Sets the PEM formatted private key.
verboseLogs
No
false
boolean
Enables verbose TLS logging.
Enables or disables TLS.
Example value: false
.
Sets the PEM formatted public certificate. Optional. If not set, it will default to ``.
Example value: -----BEGIN CERTIFICATE----- EXampLeRanDoM ... -----END CERTIFICATE-----
.
Sets the PEM formatted private key. Optional. If not set, it will default to ``.
Example value: -----BEGIN PRIVATE KEY----- ExAmPlErAnDoM ... -----END PRIVATE KEY-----
.
Enables additional logging of TLS settings and events at debug level. The information presented might be a bit too much for day to day use but can provide extra information for troubleshooting TLS configuration. Optional. If not set, it will default to false
.
Configures database settings.
It has the following fields:
host
Yes
n/a
string
Sets the name of the host to connect to.
username
No
``
string
Sets the username to authenticate as.
password
No
``
string
Sets the password to authenticate as.
database
Yes
n/a
string
Sets the database to use.
schema
No
``
string
Sets the schema to use.
TLS
No
false
boolean
Enables TLS.
params
No
{}
DBConnectionParams
Provides fine-grained control.
Sets the name of the host to connect to. A comma-separated list of host names is also accepted; each host name in the list is tried in order.
Example value: postgres:5432
.
Sets the username to authenticate as. Optional. If not set, it will default to ``.
Example value: johhnybingo
.
Sets the password to authenticate as. Optional. If not set, it will default to ``.
Example value: my-password
.
Sets the database to use.
Example value: my-database
.
Sets the schema to use. Optional. If not set, it will default to "".
Example value: my-schema
.
Enables TLS. In PostgreSQL connection string terms, setting TLS to false
corresponds to sslmode=disable
; setting TLS to true
corresponds to sslmode=verify-full
. For more fine-grained control, specify sslmode
in the params which takes precedence. Optional. If not set, it will default to false
.
Example value: true
.
Contains connection string parameters as key/values pairs. It allow fine-grained control of connection settings. The parameters can be found here: https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-PARAMKEYWORDS Optional. If not set, it will default to {}
.
Example value: {"application_name":"example"}
.
Sets the logger behaviour.
It has the following fields:
mode
Yes
n/a
string
Controls the format of the logger's output.
level
No
info
string
Controls the level of the logger.
Controls the format of the logger's output. Allowed values are text
or json
.
Controls the level of the logger. Allowed values are info
or debug
. Optional. If not set, it will default to info
.
Controls the metrics settings.
It has the following fields:
prometheusAddress
No
:9090
string
Sets the Prometheus address.
Sets the address at which Prometheus metrics are served. Optional. If not set, it will default to :9090
.
Holds the license key.
It has the following fields:
key
Yes
n/a
string
Sets the license key.
Sets the license key. An HQ key starts with "licensekey".
Accepts the Lenses EULA.
Returns all users
Creates a new user.
Sets the unique name of the new user. It must be a valid HQ resource name: it can only contain lowercase alphanumeric characters or hyphens; hyphens cannot appear at the end or start; the length is 63 characters at most.
Sets the display name of the new user. If not provided, the value of "name" will be used.
Allows attaching custom string key/values to resources. The following maxima apply:
Returns a specific user
Updates a user.
Updates a user. Absent fields are left untouched.
Updates the display name of the user.
Patches metadata. It has the following semantics:
Deletes a user.
No Content
Allows updating fields of the user profile.
Allows changes of a user's Profile. Absent fields are left untouched.
Contains the users' full name, e.g. Mary Jane Doe.
Contains the users' email address, e.g. mary.jane@doe.net. Note that this is not necessarily the same as the user's name, which often looks like an email address, but is not per se.
Assigns the given user exactly to the provided groups, ensuring they are not part of any other groups.
The name of the user.
Defines the groups a user or service account should be exactly a member of.
Adds the user or service account to the groups (specified by their names).
Removes the user or service account from the groups (specified by their names). If a group is specified in both add_to_groups as well in here, removal wins.
Sets the user or service account memberships to those groups (specified by their names) in an absolute fashion (ensures user/sa will be exactly a member of those), if provided. Cannot be combined with the add_to_groups or remove_from_groups.
Returns the currently authenticated user
Starts a session given a username/password and puts it into a cookie.
Bundles a username/password for authentication purposes.
No Content
Deletes all sessions associated with the current user.
No Content
Lists all groups
Creates a new Group.
Sets the unique name of the new group. It must be a valid HQ resource name: it can only contain lowercase alphanumeric characters or hyphens; hyphens cannot appear at the end or start; the length is 63 characters at most.
Sets the display name of the new group. If not provided, the value of "name" will be used.
Sets the description of the new group.
Lists principal names (users, service accounts) to be member of this group.
Sets the Roles that are bound to this Group by name.
Allows attaching custom string key/values to resources. The following maxima apply:
Gets a group by its name.
Updates a group.
Updates the display name of the group.
Updates the Group description, if a value is provided.
Sets the Roles that are bound to this Group to the Roles (specified by their names), if provided.
Adds the users/principals (specified by their names) to this group, if provided.
Removes the users/principals (specified by their names) from this group, if provided. If members are specified in both add_members as well in here, removal wins.
Sets the members of this group to those users/principals (specified by their names) in an absolute fashion, if provided. Cannot be combined with the add_members or remove_members fields.
Patches metadata. It has the following semantics:
Deletes a group.
No Content
Returns all roles.
Creates a new role.
Contains the fields needed to create a role.
Sets the unique name of the new role. It must be a valid HQ resource name: it can only contain lowercase alphanumeric characters or hyphens; hyphens cannot appear at the end or start; the length is 63 characters at most.
Sets the display name of the new role. If not provided, the value of "name" will be used.
Contains a list of permission statements.
Allows attaching custom string key/values to resources. The following maxima apply:
Returns a specific role.
Updates a role.
Updates a role. Absent fields are left untouched.
Updates the display name of the role.
Sets, if specififed, the new permission statements.
Patches metadata. It has the following semantics:
Deletes a role.
No Content
Assigns the given service account exactly to the provided groups, ensuring they are not part of any other groups.
The name of the service account.
Defines the groups a user or service account should be exactly a member of.
Adds the user or service account to the groups (specified by their names).
Removes the user or service account from the groups (specified by their names). If a group is specified in both add_to_groups as well in here, removal wins.
Sets the user or service account memberships to those groups (specified by their names) in an absolute fashion (ensures user/sa will be exactly a member of those), if provided. Cannot be combined with the add_to_groups or remove_from_groups.
Renews the service account's token. The current token is invalidated and a new one is generated. An optional expiration timestamp can be provided.
Determines the moment of token expiration. If not specified, the token will never expire.
Deletes a ServiceAccount.
No Content
Updates a service account.
Updates a service account. Absent fields are left untouched.
Updates the display name of the service account.
Updates the description of a service account.
Patches metadata. It has the following semantics:
Returns a specific ServiceAccount.
Creates a new ServiceAccount.
Sets the unique name of the new service account. It must be a valid HQ resource name: it can only contain lowercase alphanumeric characters or hyphens; hyphens cannot appear at the end or start; the length is 63 characters at most.
Sets | the display name of the new service account. If not provided, the value of "name" will be used.
Sets the description of the new service account.
Determines the moment of token expiration. If not specified, the token will never expire.
Allows attaching custom string key/values to resources. The following maxima apply:
Returns all ServiceAccounts.
Lists all environments
Creates a new environment.
Sets the name of the new environment. It must be a valid HQ resource name: it can only contain lowercase alphanumeric characters or hyphens; hyphens cannot appear at the end or start; the length is 63 characters at most.
Sets the display name of the new environment. If not provided, the value of "name" will be used.
Enumerates Tiers.
Allows attaching custom string key/values to resources. The following maxima apply:
Retrieves a single environment by name.
Updates an environment.
Enumerates Tiers.
Patches metadata. It has the following semantics:
Updates the display name of the environment.
Deletes an environment.
No Content
Provides Server-Sent Events (SSE) for environment updates. TODO.
Proxies HTTP to a Lenses instance. Note: this is not a regular HTTP API endpoint. The path specified here is a prefix. Everything beneath it gets proxied to the corresponding Lenses instance. Any request body and method (the GET here is only a placeholder) are accepted, as long as the Lenses API accepts it. The connection can even be upgraded to a websocket. The status code and response body are controlled by the Lenses API. This concept does not fit into the OpenAPI world at all; this definition is only here for the sake of documentation to avoid having an undocumented dark matter API.
Retrieves a list of datasets
The page number to be returned, must be greater than zero. Defaults to 1.
1
The elements amount on a single page, must be greater than zero.
25
A search keyword to match dataset, fields and description against.
name
A list of connection names to filter by. All connections will be included when no value is supplied.
A list of tag names to filter by. All tags will be included when no value is supplied.
The field to sort results by
Sorting order. Defaults to ascending
A flag to include in the search also system entities (e.g. Kafka's __consumer_offsets
topic).
Whether to search only by table name, or also to include field names/documentation (defaults to true)
Schema format. Relevant only when sourceType is SchemaRegistrySubject
Filter based on whether the dataset has records
Filter based on compacted. Relevant only when sourceType is Kafka
Get a single dataset by connection/name. While information mastered externally might be a few second out of sync with their respective sources (e.g. JMX metadata, Elasticsearch index status, etc), information mastered in Lenses's db is guaranteed to be up to date (e.g. tags, descriptions).
kafka
customer-positions
Retrieves a list of dataset tags
Get tags sorted by dataset count
user
Returns the intellisense result for a given query
Returns HQ's licence summary.
Returns HQ's licence.
Returns the backend's settings information.
Returns the backend's meta information.