Prerequisites

Permissions 

Permissions required:

• To view an SQL Processor:
  • Application permission: at least View level of the SQL Processors
  • Data Namespaces permissions: at least Show Topic for each input and output topic
• To manage an SQL Processor lifecycle (start, stop, scale and delete):
  • Application permission: at least Manage level of the SQL Processors
  • Data Namespaces permissions: at least Show Topic for each input and output topic
• To create a new SQL Processor:
  • Application permission: at least Manage level of the SQL Processors
  • Data Namespaces permissions: at least Query Topic for each input topic
  • Data Namespaces permissions: at least Insert Data for each output topic

Apart from that, when running SQL processors in CONNECT mode, permissions for given Kafka Connect cluster should be present to view/manage/create a processor running on this cluster.

Schemas must be available for Structured Data 

To support features like:

• Inference of output schemas
• Creation-time validation of input query
• Selections
• Expressions

LSQL Engine Streaming mode needs to have up-to-date schema information for all structured topics that are used as input in a given query. In this context, structured means topics that are using complex storage formats like AVRO or JSON.

INSERT INTO daily-item-purchases-stats
SELECT STREAM
    itemId
    , COUNT(*) as dailyPurchases
    , AVG(price / quantity) as average_per_unit
FROM purchases
WINDOW BY TUMBLE 1d
GROUP BY itemId;

For the above query, for example, the purchases topic will need to have a value set to a structured format and a valid schema will need to already have been configured in Lenses. In such schema, fields itemId, price and quantity must be defined, the latter two being of a numerical type.

These requirements ensure the Engine will always in a position to know what kind of data it will be working with, guaranteeing at the same time that all obvious errors are caught before a query is submitted.

Old generation processors 

Starting with Lenses 4.0, there is a new and improved SQL stream processing engine. Processors setup via earlier versions are still being managed by Lenses, after an upgrade. However, there is no support for creating SQL processors using the old engine - its syntax is deprecated; a label will appear in the user interface to highlight those specific entries.