User can be manually created in Lenses. Users can either be of type:

1. SSO, or

2. Basic Authentication

When creating a User, you can assign them groups membership.

Each user, once logged in can update their Name, Profile Photo and set an email address.

Create a User

To Create Service Account go to IAM->Users->New User, once created you can assign the user to a group.

You can also manage Users via the CLI and YAML, for integration in your CI/CD pipelines.

Admin

Full admin across all resources.

name: administrator
policy:
  - action: '*'
    resource: '*'
    effect: allow

Full access for data namespace

Allow full access for all services and resources beginning with blue.

name: blue-things
policy:
  - action:
      - iam:List*
      - iam:Get*
    resource: iam:*
    effect: allow
  - action:
      - environments:Get*
      - environments:List*
      - environments:AccessEnvironment
    resource: environments:*
    effect: allow
  - action:
      - kafka:*
      - schemas:*
      - kafka-connect:*
      - kubernetes:*
      - applications:*
    resource:
      - kafka:topic/*/*/blue-*
      - kafka:consumer-group/*/*/blue-*
      - kafka:acl/*/*/*/user/blue-*
      - schemas:schema/*/*/blue-*
      - kafka-connect:cluster/*/*
      - kafka-connect:connector/*/*/blue-*
      - sql-streaming:processor/*/*/*/blue-*
      - kubernetes:cluster/*/*
      - kubernetes:namespace/*/*/*
      - applications:external-application/*/blue-*
    effect: allow
  - action:
      - alerts:*
      - data-policies:*
    resource:
      - alerts:alert/*/*/blue-*
      - alerts:alert-event/*/*/*
      - data-policies:policy/*/blue-*
    effect: allow

Explore a data namespace

Allow read only access for topics and schemas beginning with la.

name: public-data-explorer
policy:
  - action:
      - environments:ListEnvironments
      - environments:GetEnvironmentDetails
      - environments:AccessEnvironment
    resource: environments:environment/global*
    effect: allow
  - action:
      - kafka:ListTopics
      - kafka:ListTopicDependants
      - kafka:GetTopicDetails
      - kafka:ReadTopicData
    resource: kafka:topic/*/kafka/la-*
    effect: allow
  - action:
      - schemas:ListSchemas
      - schemas:ListSchemaDependants
      - schemas:GetSchemaDetails
    resource: schemas:schema/*/*/la-*
    effect: allow

Connect Operator

Allow operators to restart connectors and list & get IAM resource only.

name: global-connector-operator
policy:
  - action:
      - iam:List*
      - iam:Get*
    resource: iam:*
    effect: allow
  - action:
      - environments:Get*
      - environments:List*
      - environments:AccessEnvironment
    resource: environments:*
    effect: allow
  - action:
      - kafka-connect:List*
      - kafka-connect:GetClusterDetails
      - kafka-connect:GetConnectorDetails
      - kafka-connect:StartConnector
      - kafka-connect:StopConnector
    resource:
      - kafka-connect:cluster/*/*
      - kafka-connect:connector/*/*/*
    effect: allow

Explict no access to production

Explicity deny access to a production environment.

name: no-access-prod
policy:
  - action: environments:AccessEnvironment
    resource: environments:environment/prod-*
    effect: deny

Developer access

Allow developers access to topics, schemas, sql processors, consumer groups, acls, quotas, connectors for us-dev.

name: us-dev-access
policy:
  - action:
      - iam:List*
      - iam:Get*
    resource: iam:*
    effect: allow
  - action:
      - environments:Get*
      - environments:List*
    resource: environments:*
    effect: allow
  - action: environments:AccessEnvironment
    resource: environments:environment/us-dev
    effect: allow
  - action:
      - kafka:*
      - schemas:*
      - kafka-connect:*
      - kubernetes:*
      - applications:*
    resource:
      - kafka:topic/us-dev/*
      - kafka:consumer-group/us-dev/*
      - kafka:acl/us-dev/*
      - kafka:quota/us-dev/*
      - schemas:schema/us-dev/*
      - kafka-connect:cluster/us-dev/*
      - kafka-connect:connector/us-dev/*
      - sql-streaming:processor/us-dev/*
      - kubernetes:cluster/us-dev/*
      - kubernetes:namespace/us-dev/*/*
      - applications:external-application/us-dev/*
    effect: allow
  - action:
      - alerts:*
      - data-policies:*
    resource:
      - alerts:alert/us-dev/*
      - alerts:alert-event/us-dev/*
      - data-policies:policy/us-dev/*
    effect: allow

Principals (Users & Service accounts) receive their permissions based on their group membership.

Roles hold a set of policies, defining the permissions. Roles are assigned to groups.

Roles provide flexibility in how you want to provide access, you can create policy that is very open or a policy that is very granular, for example allowing operators and support engineers certain permissions to restart Connectors but denying actions that would allow them to view data or configuration options.

Roles are defined at the HQ level. This allows you to control access to not only actions at HQ but at lower environment levels, and to assign the same set of permissions across your whole Kafka landscape in a central place.

A role has:

• A unique name;

• A list of Permission Statements called a Policy.

Policies

A policies have:

• One or more actions;

• One or more resource patterns that the actions apply to;

• An effect: allow or deny.

name: [policy_name]
policy: #list of actions/resources/effect
- action:
  resource:
  effect:[allow|deny]      

A policy is defined by a YAML specification.

name: blue-things
policy:
  - action:
      - iam:List*
      - iam:Get*
    resource: iam:*
    effect: allow
  - action:
      - environments:Get*
      - environments:List*
      - environments:AccessEnvironment
    resource: environments:*
    effect: allow
  - action:
      - kafka:*
      - schemas:*
      - kafka-connect:*
      - kubernetes:*
      - applications:*
    resource:
      - kafka:topic/*/*/blue-*
      - kafka:consumer-group/*/*/blue-*
      - kafka:acl/*/*/*/user/blue-*
      - schemas:schema/*/*/blue-*
      - kafka-connect:cluster/*/*
      - kafka-connect:connector/*/*/blue-*
      - sql-streaming:processor/*/*/*/blue-*
      - kubernetes:cluster/*/*
      - kubernetes:namespace/*/*/*
      - applications:external-application/*/blue-*

Actions

Actions describe a set of actions. Concrete actions can match an Action Pattern. In this text, action and action patterns are used interchangeably.

Services

Services describe the system entity that an action applies to. Services are:

1. environments

2. kafka

3. registry

4. schemas

5. kafka-connect

6. sql-streaming

7. kubernetes

8. applications

9. alerts

10. data-policies

11. governance

12. audit

13. iam

14. administration

Operations

Operation can contain a wildcard. If so, only at the end. See IAM Reference for the available operations per service.

Resources

Resources identify which resource, in a service, that the principal is allowed or denied, to perform the operation on.

If the service is provided, resource-type can be a wildcard.

Resource Path

The resource path identifies the resource within the context of a service and a resource type.

A resource-path consists of one or more segments separated by /. A segment can be a wildcard, or contain a wildcard as a suffix of a string. If a segment is a wildcard, then remaining segments do not need to be provided, and will be assumed to be wildcards as well.

Where LRN is the Lenses Resource Name

• kafka:topic/my-env/* will be expanded to kafka:topic/my-env/*/*;

• kafka:topic/my-env/my-cluster* is invalid because the Topic segment is missing, kafka:topic/my-env/my-cluster*/topic would be valid though;

• *:topic/* is invalid, the service is not provided;

• kaf*:* and kafka:top* are invalid, service and resource-type cannot contain wildcards;

• kafka:*/foo is invalid, if the resource-type is a wildcard then resource-id cannot be set.

Evaluation

A principal (user or service account) can perform an action on a resource if:

In any of the roles it receives via group membership:

• There is any matching Permission Statement that has an effect of allow;

• And there is not any matching Permission Statement that has an effect of deny.

A Permission Statement matches an action plus resource, if:

• The action matches any of the Permission Statement's Action Patterns, AND:

• The resource matches any of the Permission Statement's Resource Patterns.

An Action matches an Action Pattern (AP) if:

• The AP is a wildcard, OR:

• The Action's service equals the AP's and the AP's operation string-matches the Action's operation.

A Resource matches a Resource Pattern (RP) if:

• The RP is a wildcard, OR:

• The Resource's services equals the RP's and the RP's resource-type is a wildcard, OR:

• The Resource's service and types equals that of the RP and resource-ids match. Resource-ids are matched by string-matching each individual segment. If the RP has a trailing wildcard segment, the remaining segments are ignored.

A string s matches p if:

• They equal character by character.

• If s or p has more non-wildcard characters than the other they don't match;

• If p contains a * suffix, any remaining characters in s are ignored.

Order of items in any collection is irrelevant during evaluation. Collections are considered sets rather than ordered lists. The following are equivalent:

• Order of Resource Patterns does not matter

• Order of Permission Statements does not matter

• Order of Roles does not matter

• Order of Groups does not matter

Examples

In the examples we're not too religious about strict JSON formatting.

Broad Allow + Specific Deny

Given:

policy:
  - effect: allow
    resource: kafka:topic/my-env/*/*
    action: ReadKafkaData
  - effect: deny
    resource: kafka:topic/*/*/forbidden-topic
    action: ReadKafkaData

A principal:

• Can ReadKafkaData on kafka:topic/my-env/the-cluster/some-topic because it is allowed and not denied;

• Cannot DeleteKafkaTopic on kafka:topic/my-env/the-cluster/some-topic because there is no allow;

• Cannot ReadKafkaData on kafka:topic/my-env/the-cluster/forbidden-topic because while it is allowed the deny kicks in.

Multiple Resources I

Given:

policy:  
  - effect: allow
    resource: [*, kafka:topic/my-cluster/*]
    action: ReadKafkaData

A principal:

• Can ReadKafkaData on kafka:topic/someone-else-cluster/their-topic because the resource matches *.

Note that here the matching can be considered "most permissive".

Multiple Resources II

Given:

policy:
  - effect: allow
    resource: [kafka:topic/my-cluster/my-topic-1, kafka:topic/my-cluster/my-topic-2]
    action: ReadKafkaData

A principal:

• Can ReadKafkaData on kafka:topic/my-cluster/my-topic-1 and kafka:topic/my-cluster/my-topic-2 because the resources match, but cannot ReadKafkaData on kafka:topic/my-cluster/my-topic-3.

Lenses IAM is built around Roles. Roles contain policies and each policy defines a set of actions a user is allow to take.

Roles are then assigned to groups.

Role Policies

The Lenses policies are resource based. They are YAML based documents attached to a resource.

Each policy has:

1. Action

2. Resource

3. Effect

Action

The action describes the action or verb that a user can perform. The format of the action is

[entity type]:action

For example to list topics in Kafka

policy
  - action:
    - kafka:ListTopics

Resource

To restrict access to resources, for example, only list topics being with red we can used use the resource field.

Effect

Effect is either allow the action on the resource or deny. If allow is not set the action will be denied and if any policy for a resource has a deny effect it takes precedence.

Create a Role

To Create Service Account go to IAM->Roles->New Role.

You can also manage Users via the CLI and YAML, for integration in your CI/CD pipelines.

➜  hq roles
Manage Roles.

Usage:
  hq roles [command]

Available Commands:
  create      Creates a new role.
  delete      Deletes a role.
  get         Returns a specific role.
  list        Returns all roles.
  metadata    Manages role metadata.
  update      Updates a role.

Service accounts are intended for programmatic access to Lenses.

Each service account has a key that is used to authenticate and identify the service account.

In addition you can set:

1. Description

2. Resource name - Must be unique across Lenses.

3. Key expiry

4. Regenerate the key

Key expiry can be 7, 30, 60, 90 days, 1 year or a custom expiration or no expiration at all.

Creating a Service Account

To Create Service Account go to IAM->Service Accounts->New Service Account, once created you can then assign service accounts to groups.

You can also manage Users via the CLI and YAML, for integration in your CI/CD pipelines.

➜  hq service-accounts
Manage ServiceAccounts.

Usage:
  hq service-accounts [command]

Aliases:
  service-accounts, sa

Available Commands:
  create      Creates a new ServiceAccount.
  delete      Deletes a ServiceAccount.
  get         Returns a specific ServiceAccount.
  list        Returns all ServiceAccounts.
  metadata    Manages service-account metadata.
  renew-token Renews the service account's token. The current token is invalidated and a new one is generated. An optional expiration timestamp can be provided.
  set-groups  Assigns the given service account exactly to the provided groups, ensuring they are not part of any other groups.
  update      Updates a service account.

Groups are a collection of users, service accounts and roles.

Assigning Users to Groups

Users can be assign to Groups in two ways:

1. Manual

2. Linked from the groups provided by your SSO provider

This behaviour can be toggled in the organizational settings of your profile. To control the default set the following in the config.yaml for HQ.

users_group_membership_management_mode: [manual|sso]

Groups can be defined with the following metadata:

1. Colour

2. Description

Each group has a resource that unique identifies it across an HQ installation.

Create a Group

To Create Group go to IAM->Groups->New Group, create the group, assign members, service accounts and roles.

You can also manage Users via the CLI and YAML, for integration in your CI/CD pipelines.

➜  hq groups
Manage Groups.

Usage:
  hq groups [command]

Aliases:
  groups, grp

Available Commands:
  create      Creates a new Group.
  delete      Deletes a group.
  get         Gets a group by its name.
  list        Lists all groups
  metadata    Manages group metadata.
  update      Updates a group.

Administration

service: administration

Resource Syntax

• admin:connection:${Environment}/${ConnectionType}/${Connection}

• admin:license:${Environment}

• admin:lenses-logs:${Environment}

• admin:lenses-configuration:${Environment}

• admin:setting:${Setting}

Applications

service: applications

Resource Syntax

Alerts

service: alerts

Resource Syntax

• alerts:alert:${Environment}/${AlertType}/${Alert}

• alerts:rule:${Environment}/Infrastructure/KafkaBrokerDown

• alerts:rule:${Environment}/DataProduced/red-app-going-slow

Audits

service: audit

Resource Syntax

• audit:log:${Environment}

• audit:channel:${Environment}/${AuditChannelType}/${AuditChannel}

Data Policies

service: data-policies

Resource Syntax

• data-policies:policy/${Environment}/${Policy}

Environments

service: environments

Resource Syntax

• environments:environment/${Environment}

Governance

service: governance

Resource Syntax

• governance:request:${Environment}/${ActionType}/*

• governance:rule:${Environment}/${RuleCategory}/*

IAM

service: iam

Resource Syntax

• iam:role:${Role}

• iam:group:${Group}

• iam:user:${Username}

• iam:service-account:${ServiceAccount}

Kafka Connect

service: kafka-connect

Resource Syntax

• kafka-connect:connector:${Environment}/${KafkaConnectCluster}/${Connector}

• kafka-connect:cluster:${Environment}/${KafkaConnectCluster}

name: global-connector-operator
policy:
  - action:
      - iam:List*
      - iam:Get*
    resource: iam:*
    effect: allow
  - action:
      - environments:Get*
      - environments:List*
      - environments:AccessEnvironment
    resource: environments:*
    effect: allow
  - action:
      - kafka-connect:List*
      - kafka-connect:GetClusterDetails
      - kafka-connect:GetConnectorDetails
      - kafka-connect:StartConnector
      - kafka-connect:StopConnector
    resource:
      - kafka-connect:cluster/*/*
      - kafka-connect:connector/*/*/*
    effect: allow

Kafka

service: kafka

Resource Syntax

• kafka:topic:${Environment}/${KafkaCluster}/${Topic}

• kafka:acl:${Environment}/${KafkaCluster}/${AclResourceType}/* or kafka:acl:${Environment}/${KafkaCluster}/${AclResourceType}/${PrincipalType}/${Principal}

• kafka:quota:${Environment}/${KafkaCluster}/${QuotaType}/* or

• kafka:quota:${Environment}/${KafkaCluster}/clients

• kafka:quota:${Environment}/${KafkaCluster}/users-default

• kafka:quota:${Environment}/${KafkaCluster}/client/${ClientID}

• kafka:quota:${Environment}/${KafkaCluster}/user/${Username}

• kafka:quota:${Environment}/${KafkaCluster}/user/${Username}/client/${ClientID}

• kafka:quota:${Environment}/${KafkaCluster}/user-client/${Username}/${ClientID}

• kafka:quota:${Environment}/${KafkaCluster}/user/${Username}/client/*

• kafka:quota:${Environment}/${KafkaCluster}/user-all-clients/${Username}

name: example
policy:
  - action:
      - kafka:ListTopic
      - kafka:GetTopicDetails 
    resource: 
      - kafka:topic:my_env/kafka/my_topic

Kubernetes

service: kubernetes

Resource Syntax

• kubernetes:cluster:${Environment}/${KubernetesCluster}

• kubernetes:namespace:${Environment}/${KubernetesCluster}/${KubernetesNamespace}

Registry

service: registry

Resource Syntax

• schemas:registry:${Environment}/${SchemaRegistry}

Schemas

service: schemas

Resource Syntax

• schemas:schema:${Environment}/${SchemaRegistry}/${Schema}

SQL Streaming

service: sql-streaming

Resource Syntax

• sql-streaming:sql-processor:${Environment}/${KubernetesCluster}/${KubernetesNamespace}/${SqlProcessor}

• For IN_PROC processors sql-streaming:sql-processor:${Environment}/lenses-in-process/default/${SqlProcessor}