Kafka 101

Kafka Concepts

After downloading the binary, unzip it and go to the main folder that is unzipped

1$ tar -xzf kafka_zipped_version_here.tgz
2$ cd kafka_unzipped_folder

Start Zookeeper server:

1$ bin/zookeeper-server-start.sh config/zookeeper.properties

On another terminal, start Kafka server:

1$ bin/kafka-server-start.sh config/server.properties

To create a topic on the zookeeper, with 3 partition and single replica factor

1$ kafka-topics zookeeper 127.0.0.1:2181 --topic first_topic --create --partitions 3 --replication-factor 1

Recall that the replica factor cannot be greater than the brokers, if you have a single broker, you cannot have a replica broker...

To see the details of the first_topic

1$ kafka-topics zookeeper 127.0.0.1:2181 --topic first_topic --describe

To list the topics

1$ kafka-topics zookeeper 127.0.0.1:2181 --list

To delete the topic, other than Windows systems

1$ kafka-topics zookeeper 127.0.0.1:2181 --topic first_topic --delete

Now we created topics, let's send datastreams to the topics

First of all, let's get the help file

When you are in the installation folder in your linux terminal, type:

1$ bin/kafka-console-producer.sh

This shows the help concepts, note that --broker-list and --topic are required fields to send data to the brokers

To send data on the first_topic just type

1$ bin/kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic first_topic 
2>

when you hit enter, a caret will appear, and you can type your messages to end the message stream, hit Ctrl + C

Now we created data producers, let's consume them

First of all, let's get the help file

When you are in the installation folder in your linux terminal, type:

1$ bin/kafka-console-consumer.sh

Note that the bootstrap-server is required, while not noted as required, the topic is also used

to consume the data from the topic

1$ bin/kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092 --topic first_topic

when you hit the enter, there is a caret that appears but we cannot read any message from the first topic

This is because, we set the consumer to read the messages realtime from now on

So, if you open a new console and start feeding messages to the first_topic, you will see the message appearing on the consumer terminal

Let's do it, don't close the consumer console above but start another console

1$ bin/kafka-console-producer --broker-list 127.0.0.1:9092 --topic first_topic 
2> my first datastream

should appear on the consumer terminal as well, voila!

To read all messages from the beginning, just add the --from-beginning flag to the end

1$ bin/kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092 --topic first_topic --from-beginning