Kafka Setup Instructions:
1. Download opensource Kafka from official website and extract the binary:
a. go to https://kafka.apache.org/downloads
b. download the recommended version from Binary downloads.
eg. Scala 2.13 - kafka_2.13-3.5.1.tgz (asc, sha512)
c. Extract using 7-zip and rename it to kafka
*************************
2. Do some configuration:
Update server.properties and zookeeper.properties files located in config directory.
a. update server.properties::
# A comma separated list of directories under which to store log files
log.dirs=C:/IdeaLab/kafka/kafka-logs #this is the directory where logs will be residing
b. update zookeeper.properties::
# the directory where the snapshot is stored.
dataDir=C:/IdeaLab/kafka/zookeeper-data #this will be the data directory for our zookeeper
*************************
3. Kickoff zookeeper and kafka server:
a. start zookeeper:
C:\IdeaLab\kafka\bin\windows\zookeeper-server-start.bat C:\IdeaLab\kafka\config\zookeeper.properties
####NOTE: Don't close the zookeeper terminal window. Instead just open new terminal window to start kafka server.
b. kickoff the kafka server:
when you open the terminal inside kafka directory, you don't need to give the full path. You can just give .\bin\... as well.
.\bin\windows\kafka-server-start.bat .\config\server.properties
OR
C:\IdeaLab\kafka\bin\windows\kafka-server-start.bat C:\IdeaLab\kafka\config\server.properties
*************************
4. Produce the data and consume it from the consumer function.
a. we need to first create a test topic and push some data on it.
C:\IdeaLab\kafka\bin\windows>kafka-topics.bat --create --bootstrap-server localhost:9092 --topic vk_kafka_topic
Created topic vk_kafka_topic
b.
C:\IdeaLab\kafka\bin\windows> .\kafka-console-consumer.bat --topic vk_kafka_topic --bootstrap-server localhost:9092 --from-beginning
