Blockchain

A Kafka Connector for listening to blockchain events.

KCQL support 

KCQL is not supported.

Concepts 

A Kafka Connector to hook into the live streaming providing a real-time feed for new bitcoin blocks and transactions provided by www.blockhain.info The connector subscribes to notification on blocks, transactions or an address and receives JSON objects describing a transaction or block when an event occurs. This JSON is then pushed via Kafka connect to a Kafka topic and therefore can be consumed either by a Sink or a live stream processor.

Since is a direct WebSocket connection the source will only ever use one connector task at any point. There is no point spawning more and then have duplicate data.

One thing to remember is the subscription API from blockchain doesn’t offer an option to start from a given timestamp. This means if the connect worker is down then you will miss some data.

Quickstart 

Launch the stack 

1. Copy the docker-compose file.
2. Bring up the stack.

export CONNECTOR=blockchain
docker-compose up -d fastdata

Start the connector 

If you are using Lenses, login into Lenses and navigate to the connectors page , select Blockchain as the source and paste the following:

name=blockchain
connector.class=com.datamountaineer.streamreactor.connect.blockchain.source.BlockchainSourceConnector
connect.blockchain.source.kafka.topic=blockchain
tasks.max=1

To start the connector without using Lenses, log into the fastdatadev container:

docker exec -ti fastdata /bin/bash

and create a connector.properties file containing the properties above.

Create the connector, with the connect-cli :

connect-cli create blockchain < connector.properties

Wait a for the connector to start and check its running:

connect-cli status blockchain

Check for records in Kafka 

Check the records in Lenses or with via the console:

kafka-avro-console-consumer \
    --bootstrap-server localhost:9092 \
    --topic blockchain \
    --from-beginning

Clean up 

Bring down the stack:

docker-compose down

Options