-->
  • Recent Articles

    How to install kafka - Standalone Installation

    How to install kafka - Standalone Installation


    How to install kafka - Standalone Installation

    As part of the this tutorial, we will be setting up Apache Kafka in a single node. Apache Kafka is an open-source message broker project developed by the Apache Software Foundation written in Scala. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds.

    Download Kafka


    Download Latest Kafka From the official Apache Kafka Website


    [root@monitor tmp]# wget http://apachemirror.wuchna.com/kafka/2.4.0/kafka_2.11-2.4.0.tgz
    --2020-02-28 14:38:08--  http://apachemirror.wuchna.com/kafka/2.4.0/kafka_2.11-2.4.0.tgz
    Resolving apachemirror.wuchna.com (apachemirror.wuchna.com)... 159.65.154.237
    Connecting to apachemirror.wuchna.com (apachemirror.wuchna.com)|159.65.154.237|:80... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 70057083 (67M) [application/x-gzip]
    Saving to: ‘kafka_2.11-2.4.0.tgz’

    100%[==========================================================================================>] 70,057,083  19.6MB/s   in 4.5s

    2020-02-28 14:38:12 (15.0 MB/s) - ‘kafka_2.11-2.4.0.tgz’ saved [70057083/70057083]

    [root@monitor tmp]#

    Install Java


    [root@monitor tmp]# yum install java-1.8.0-openjdk
    [root@monitor tmp]# yum install java-1.8.0-openjdk-devel

    Untar the tarball

    [root@monitor tmp]# tar xzvf kafka_2.11-2.4.0.tgz
    kafka_2.11-2.4.0/
    kafka_2.11-2.4.0/LICENSE
    kafka_2.11-2.4.0/NOTICE
    kafka_2.11-2.4.0/bin/
    kafka_2.11-2.4.0/bin/kafka-delete-records.sh
    kafka_2.11-2.4.0/bin/trogdor.sh
    kafka_2.11-2.4.0/bin/kafka-preferred-replica-election.sh
    kafka_2.11-2.4.0/bin/connect-mirror-maker.sh
    kafka_2.11-2.4.0/bin/kafka-console-consumer.sh

    Start Zookeeper

    Kafka relies on Zookeeper to for configuration information.

    [root@monitor tmp]# bin/zookeeper-server-start.sh config/zookeeper.properties

    Start the Kafka Servics


    [root@monitor tmp]# kafka_2.11-2.4.0/bin/kafka-server-start.sh kafka_2.11-2.4.0/config/server.properties
    [2020-02-28 14:41:17,097] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
    [2020-02-28 14:41:17,386] INFO Registered signal handlers for TERM, INT, HUP (org.apache.kafka.common.utils.LoggingSignalHandler)
    [2020-02-28 14:41:17,386] INFO starting (kafka.server.KafkaServer)
    [2020-02-28 14:41:17,387] INFO Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer)
    [2020-02-28 14:41:17,431] INFO [ZooKeeperClient Kafka server] Initializing a new session to localhost:2181. (kafka.zookeeper.ZooKeeperClient)

     Create Topic

    Kafka partitions incoming messages for a topic, and assigns those partitions to the available Kafka brokers. The number of partitions is configurable and can be set per-topic and per-broker. The following command will create a topic test with one partition

    [root@monitor kafka_2.11-2.4.0]# bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
    Created topic test.
    [root@monitor kafka_2.11-2.4.0]#

     List Topics

    To make sure wheather the topic is created successfully, we can list out the topics hosted by the broker with the following command.

    [root@monitor kafka_2.11-2.4.0]# bin/kafka-topics.sh --list --zookeeper localhost:2181
    test
    [root@monitor kafka_2.11-2.4.0]#

    Check more Kafka Articles:

    Start Producer

    Producers publish messages to Kafka topics and are able to choose which topic, and which partition within the topic, to publish the message to. we will start a producer no

    [root@monitor kafka_2.11-2.4.0]# bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
    >test
    >this is manas
    >who are you
    >how are you
    >^CYou have new mail in /var/spool/mail/root
    [root@monitor kafka_2.11-2.4.0]#

    Start Consumer

    Consumers subscribe to these topics and consume the messages. Lets consume the messages with the following command

    [root@monitor kafka_2.11-2.4.0]# bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
    test
    this is manas
    who are you
    how are you
    ^CProcessed a total of 4 messages
    You have new mail in /var/spool/mail/root
    [root@monitor kafka_2.11-2.4.0]#

    No comments