> For the complete documentation index, see [llms.txt](https://docs.lenses.io/latest/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.lenses.io/latest/deployment/configuration/security-hardening.md). # Security Hardening Recommended controls for running Lenses in production on Kubernetes: least-privilege database roles, network segmentation, scoped RBAC, and resource governance. This page collects the controls we recommend for a production Lenses deployment on Kubernetes. Each one has been applied to a full Lenses installation — HQ, Agents, SQL Processors, K2K and MCP — and the deployment exercised afterwards to confirm it still works. None of these controls are required for Lenses to run — they reduce the blast radius if something else in your environment is compromised, and most Kubernetes and Postgres administrators will recognise them as standard defence-in-depth practice. {% hint style="warning" %} **Apply these controls before, or alongside, the workloads they govern — not afterwards to a running cluster.** Several of the controls on this page only take effect at a particular moment: a `LimitRange` is enforced when a pod is **created**, and a `NetworkPolicy` filters connections when they are **established**. Applying them to an already-running deployment therefore changes nothing immediately. Existing pods keep their original resources and their already-open connections, so the cluster looks healthy and the controls look satisfied — even where they are actually unsatisfiable. The mismatch surfaces later, all at once, the next time pods are recreated: a node pool upgrade, a scale-up, a Helm release, a drain. If you are adding these controls to an existing installation, roll the workloads afterwards and confirm they come back healthy, so any conflict surfaces while you are watching rather than during an unattended cluster upgrade. {% endhint %} ## 1. Least-privilege database roles Lenses HQ and each Agent need a database to store their own metadata. Give each component its own database and a dedicated, non-superuser role that owns only that database: ```sql CREATE ROLE lenses_hq WITH LOGIN NOSUPERUSER NOCREATEDB NOCREATEROLE NOREPLICATION PASSWORD ''; CREATE DATABASE lenses_hq OWNER lenses_hq; REVOKE ALL ON DATABASE lenses_hq FROM PUBLIC; GRANT CONNECT ON DATABASE lenses_hq TO lenses_hq; ``` Repeat with a different role and database per component (HQ, each Agent) rather than sharing one role across all of them — a credential leaked from one component then only exposes that component's data. {% hint style="danger" %} **Never over-grant permissions to a Lenses database role** — for example `SUPERUSER`, or membership of `pg_execute_server_program`. Privileges at that level allow arbitrary command execution on the database server, turning an exposed Lenses credential into a compromise of the database host rather than of one component's data. {% endhint %} ## 2. Network segmentation Apply a default-deny `NetworkPolicy` for both ingress and egress in every namespace running Lenses components, then add narrow allow rules for the traffic each component actually needs. ```yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny namespace: lenses spec: podSelector: {} policyTypes: ["Ingress", "Egress"] ``` ### DNS keeps working, even under default-deny DNS resolution has to be explicitly allowed or nothing in the namespace can resolve a hostname, including Lenses' own connections to Kafka, Schema Registry, and Postgres. {% hint style="warning" %} **A `namespaceSelector` rule for `kube-system` may not be sufficient on its own.** Some clusters run a node-local DNS cache whose pods use the host network, so their addresses are node IPs rather than pod IPs and no `namespaceSelector` peer will match them. If DNS lookups from inside the namespace hang while other traffic works, add explicit `ipBlock` rules for your cluster's DNS addresses alongside the `namespaceSelector` rule. Check your Kubernetes provider's documentation for the addresses in use. {% endhint %} ```yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-dns namespace: lenses spec: podSelector: {} policyTypes: ["Egress"] egress: - to: - namespaceSelector: matchLabels: kubernetes.io/metadata.name: kube-system ports: - port: 53 protocol: UDP - port: 53 protocol: TCP - to: # Your cluster's node-local DNS cache address, if it runs one # (commonly a link-local address such as 169.254.20.10) — confirm # the value for your cluster. - ipBlock: cidr: 169.254.20.10/32 ports: - port: 53 protocol: UDP - port: 53 protocol: TCP - to: # The kube-dns Service ClusterIP — check yours with: # kubectl -n kube-system get svc kube-dns -o jsonpath='{.spec.clusterIP}' - ipBlock: cidr: /32 ports: - port: 53 protocol: UDP - port: 53 protocol: TCP ``` ### Bound egress to the internet If a namespace's workloads need general outbound access (for example, license checks or reaching an external Kafka/Schema Registry), allow it explicitly rather than leaving egress unrestricted, and exclude the cloud metadata endpoint and private address ranges the workload has no legitimate reason to reach: ```yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-internet-egress namespace: lenses spec: podSelector: {} policyTypes: ["Egress"] egress: - to: - ipBlock: cidr: 0.0.0.0/0 except: - 169.254.0.0/16 # link-local, includes the cloud metadata endpoint - 10.0.0.0/8 - 172.16.0.0/12 - 192.168.0.0/16 ``` This is intentionally broader than just excluding the metadata IP: blocking the whole link-local and RFC1918 ranges also stops a workload reaching other internal services it was never meant to talk to, not only the metadata endpoint. {% hint style="warning" %} **The Agent needs to reach the Kubernetes API server, and on many clusters that is an RFC1918 address.** The policy above excludes `10.0.0.0/8`, `172.16.0.0/12` and `192.168.0.0/16`, so applied as-is in the Agent's namespace it blocks the API server too — and SQL Processor and K2K deployment stops working. Add a narrow allow rule for the API server alongside it. Find the address with: ```bash kubectl -n default get endpoints kubernetes -o jsonpath='{.subsets[*].addresses[*].ip}' ``` then permit exactly that, rather than re-opening a whole private range: ```yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-k8s-api-egress namespace: lenses spec: podSelector: {} policyTypes: ["Egress"] egress: - to: - ipBlock: cidr: /32 # e.g. 172.16.0.2/32 on a private GKE cluster ports: - port: 443 protocol: TCP ``` NetworkPolicies are additive, so this rule re-permits only that one address and port while the exclusions above continue to apply to everything else. **On managed clusters this address is not permanent.** A `/32` pinned today can stop matching after a control-plane upgrade, repair, or resize — GKE rotates the control-plane address, and on EKS the API server is reached through ENIs whose addresses change as they are replaced. The symptom is the one described below: nothing breaks until pods restart, and then the Agent cannot reach the API server. Either re-check the endpoint after any control-plane change, or pin the range the provider guarantees instead of a single address — on GKE that is the control-plane CIDR: ```bash gcloud container clusters describe --region \ --format='value(privateClusterConfig.masterIpv4CidrBlock)' # a /28 ``` {% endhint %} ### Component-to-component traffic Beyond DNS and internet egress, add narrow allow rules for the specific flows your deployment needs — for example, Agent to Kafka/Schema Registry/Kafka Connect, Agent to HQ's gRPC port, and HQ/Agent to PostgreSQL. Scope each rule to the specific port and, where possible, the specific namespace or pod label of the peer, rather than allowing a namespace's pods to reach each other on every port. {% hint style="warning" %} **Test a new `NetworkPolicy` by restarting the affected pods, not by watching the running ones.** A `NetworkPolicy` filters **new** connections. Connections that were already established when you applied it keep working, because they are tracked in the kernel's connection-tracking table. A policy that is missing a required port can therefore look completely healthy — sometimes for days — and then break the next time a pod restarts, a node is drained, or a cluster upgrade recreates the workloads. After applying or changing a policy, restart the affected pods and confirm they come back healthy. Otherwise you are only testing the connections the policy never had to authorise. {% endhint %} ## 3. Namespace-scoped RBAC for Kubernetes-deployed apps Both **SQL Processors** and **K2K pipelines** are deployed by the Agent as Kubernetes workloads, using the Agent's own ServiceAccount. Scope that ServiceAccount with namespace-scoped `Role`/`RoleBinding` resources — never a `ClusterRole` — so the Agent can only deploy into, and read information from, the namespace(s) you explicitly allow. A `Role`/`RoleBinding` pair grants access in **one** namespace. If you deploy Processors into more than one, create a pair in **each** target namespace and bind them all to the Agent's ServiceAccount — that is still narrower than a `ClusterRole`, because the set of namespaces stays one you chose explicitly. Whichever namespaces you pick, list the same ones in `lenses.kubernetes.namespaces` (see the warning below). See [SQL Processor Deployment](/latest/deployment/configuration/agent/sql-processor-deployment.md#namespace-scoped-rbac-recommended) for the full `Role`/`RoleBinding` example and Helm values. The same RBAC also governs K2K pipeline deployments — there is no separate permission model for K2K. {% hint style="info" %} Namespace-scoped RBAC only works if you also set `lenses.kubernetes.namespaces` to the same namespace(s). Without it, the Agent's Kubernetes watchers try to list/watch cluster-wide and get denied by the scoped `Role`. See the linked page for the exact configuration — this is a common step to miss, and granting a `ClusterRole` to "fix" the resulting error reopens the cross-namespace access the scoped `Role` was meant to prevent. {% endhint %} ## 4. Resource governance Set a `ResourceQuota` and a `LimitRange` on every namespace where the Agent deploys SQL Processors or K2K pipelines. Without them, a processor or pipeline created through the API can request as much CPU and memory as the cluster will allow. ```yaml apiVersion: v1 kind: ResourceQuota metadata: name: lenses-quota namespace: lenses spec: hard: requests.cpu: "8" requests.memory: 16Gi limits.cpu: "16" limits.memory: 32Gi pods: "40" --- apiVersion: v1 kind: LimitRange metadata: name: lenses-limits namespace: lenses spec: limits: - type: Container # `max` must admit the Agent's own limit, not just Processor pods. The # chart's default Agent limit is 5Gi, so a 4Gi max here would reject the # Agent pod outright -- exactly the failure described below. max: cpu: "2" memory: 8Gi default: cpu: 500m memory: 512Mi defaultRequest: cpu: 100m memory: 256Mi ``` The `LimitRange` `max` is what actually matters here: it caps any single processor or pipeline container regardless of what resource request is submitted through the API, so it bounds the damage even if the `ResourceQuota` values above are set too high. Treat the numbers above as a starting point, not a fixed recommendation — measure your namespace's current CPU/memory usage under normal load and size the quota with headroom above it, then tighten the `LimitRange` `max` to the largest single processor or pipeline you expect to run. {% hint style="danger" %} **A `LimitRange` applies to every container in the namespace — including the Agent itself, not just the workloads it deploys.** The Agent's own memory limit must fit inside `max`. The Agent needs considerably more headroom than a SQL Processor: it holds Kafka consumer buffers for topic browsing, SQL query results, and schema and metadata caches for every topic in its environment. If `max` is lower than the Agent's configured limit, the Deployment's ReplicaSet cannot create the pod at all, and Kubernetes reports: ``` Error creating: pods "lenses-agent-..." is forbidden: maximum memory usage per Container is 4Gi, but limit is 5Gi ``` This failure is easy to misread, because there is **no pod** — so there are no logs to inspect and nothing crash-looping. The Deployment simply sits at `0/1` with `0` replicas. The rejection is visible only in the events: ```bash kubectl -n lenses get events --field-selector reason=FailedCreate kubectl -n lenses describe deploy # look for ReplicaFailure ``` Either raise `max` above the Agent's limit, or lower the Agent's limit to fit — but if you raise `max`, keep the per-processor cap where you want it using `lensesAgent.sql.memLimit` (see [Helm — Agent](/latest/deployment/installation/helm/agent.md)), which is applied by the Agent to every SQL Processor pod it creates and is independent of the `LimitRange`. {% endhint %} {% hint style="warning" %} **`default` and `defaultRequest` are injected into any container that does not state its own resources.** A container that declares no resources of its own inherits the `LimitRange` `default` as its limit. For a JVM-based component that sizes its heap from the container memory limit rather than from an explicit `-Xmx`, that means a Kubernetes setting silently determines the heap — and a small `default` can leave a heap too small for the process to start. Declare resources explicitly on the Lenses workloads in the namespace, and set the heap explicitly rather than letting it be derived from the container limit. That removes the coupling, so a later change to a memory limit cannot resize a heap as a side effect. {% endhint %} ## 5. Cross-environment K2K deployments A K2K pipeline runs as a single Kubernetes Deployment, in a single namespace, and Lenses requires the Kafka connection credentials for **both** the source and target clusters to be available in that same namespace. If you use K2K to replicate between two environments with different trust levels — for example, reading from production into a lower-trust development or staging environment — this means the higher-trust environment's Kafka credentials will need to exist inside the lower-trust environment's namespace for the pipeline to work. Apply the same access controls, RBAC scoping, and monitoring to that namespace as you apply to the source environment, or avoid routing a K2K pipeline across a trust boundary you are not willing to extend. ## Related pages * [SQL Processor Deployment](/latest/deployment/configuration/agent/sql-processor-deployment.md) — namespace-scoped RBAC and `lenses.kubernetes.namespaces` in full * [Kafka ACLs](/latest/deployment/configuration/agent/kafka-acls.md) — minimal Kafka permissions for the Agent * [Kafka Connect](/latest/deployment/configuration/agent/automation/kafka-connect.md#keep-secrets-out-of-connector-configuration) — keep secrets out of connector configuration with config providers, and limit who can view connectors * [Agent Database](/latest/deployment/configuration/agent/database.md) — configuring the Agent's database connection * [Database Migration](/latest/deployment/upgrade-to-lenses-6/database-migration.md) — least-privilege roles when migrating from Lenses 5 --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.lenses.io/latest/deployment/configuration/security-hardening.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.