View the latest documentation 5.5
Lenses works out of the box with any messages in AVRO, JSON, XML and primitive formats.
When using messages with other formats like Google Protobuf or Thrift, you need to provide a deserialization envelop or else a serde. A serde handles reading raw bytes into a meaningful data structure and translating data into raw bytes based on the storage format.
serde
For example a Kafka topic contains GPS coordinates, including a latitude and a longitude stored as double. Each “latitude:longitude” string, such as “45.623412:10.419433” is converted to UTF8 bytes, and published to a topic in Protobuf format.
A serde can enable to decode/encode this format. This can be achieved via a simple API. First use the following library dependency:
<dependency> <groupId>com.landoop</groupId> <artifactId>lenses-serde</artifactId> <version>1.0.3</version> </dependency>
And check the following Java code that implements a Serde interface that has two methods. The return type of the deserializer is a GenericRecord with the two fields parsed from the bytes.
GenericRecord
In this simple example, we do not worry about error handling, such as ensuring that the tokens have valid double values.
public class LatLngSerde implements Serde { private Schema schema = SchemaBuilder.builder() .record("lat_lng") .fields() .requiredDouble("lat") .requiredDouble("lng") .endRecord(); @Override public Schema getSchema() { return schema; } @Override public Serializer serializer(Properties properties) { return null; } @Override public Deserializer deserializer(Properties properties) { return new Deserializer() { @Override public GenericRecord deserialize(byte[] bytes) throws IOException { String data = new String(bytes); String[] tokens = data.split(":"); double lat = Double.parseDouble(tokens[0]); double lng = Double.parseDouble(tokens[1]); GenericRecord record = new GenericData.Record(schema); record.put("lat", lat); record.put("lng", lng); return record; } @Override public void close() throws IOException { } }; } }
On this page