View the latest documentation 5.5
Null values are used as the way to express a value that isn’t yet known.
Null values can be found in the data present in existing sources, or they can be the product of joining data using non-inner joins.
The schema of nullable types is represented as a union of the field type and a null value:
{ "type": "record", "name": "record", "namespace": "example", "doc": "A schema representing a nullable type.", "fields": [ { "name": "property", "type": [ "null", "double" ], "doc": "A property that can be null or a double." } ] }
Working with null values can create situations where it’s not clear what the outcome of an operation is.
One example of this would be the following:
null * 1 = ? null + 1 = ?
Looking at the first two expressions, one may be tempted to solve the problem above by saying “null is 1 when multiplying and 0 when summing” meaning the following would be the evaluation result:
null * 1 = 1 null + 1 = 1
Rewriting the third expression applying the distributive property of multiplication however shows that the rule creates inconsistencies:
(null + 1) * null = (null + 1) * null <=> null * null + 1 * null = (null + 1) * null <=> 1 * 1 + 1*1 = (0 + 1) * 1 <=> 1 + 1 = 1 * 2 <=> 1 = 2 //not valid
With the intent of avoiding scenarios like the above where a computation may have different results based on the evaluation approach taken, most operations in lenses do not allow operations to use nullable types.
Lenses provides the following tools to address nullability in a flexible way:
COALESCE(nullable_fieldA, nullable_fieldB, 0)
AS_NON_NULLABLE(nullable_field)
CASE WHEN a_nullable_field IS NULL THEN 0 ELSE AS_NON_NULLABLE(a_nullable_field) END
The AS_NULLABLE function is the inverse transformation of the AS_NON_NULLABLE version. This function allows a non nullable field type to be transformed into a nullable type. It can be used to insert data into existing topics where the schema of the target field is nullable.
AS_NULLABLE
AS_NON_NULLABLE
On this page