• Recent Articles

    How to install Kafka Manager for Managing kafka cluster

    How to install  Kafka Manager for Managing kafka cluster

    In this article, we are going to set up the Kafka management software to manage and overview our cluster. Kafka manager developed by at Yahoo. It is now called  CMAK (Cluster Manager for Apache Kafka, previously known as Kafka Manager).

    This application support following tasks.

    • Manage multiple clusters
    • Easy inspection of cluster state (topics, consumers, offsets, brokers, replica distribution, partition distribution)
    • Run preferred replica election
    • Generate partition assignments with option to select brokers to use
    • Run reassignment of partition (based on generated assignments)
    • Create a topic with optional topic configs ( has different configs than 0.8.2+)
    • Delete topic (only supported on 0.8.2+ and remember set delete.topic.enable=true in broker config)
    • Topic list now indicates topics marked for deletion (only supported on 0.8.2+)
    • Batch generate partition assignments for multiple topics with option to select brokers to use
    • Batch run reassignment of partition for multiple topics
    • Add partitions to existing topic
    • Update config for existing topic
    • Optionally enable JMX polling for broker level and topic level metrics.
    • Optionally filter out consumers that do not have ids/ owners/ & offsets/ directories in zookeeper.

    Prerequisites : 

    • Apache Zookeeper
    • Apache Kafka
    • Java 8 
    • SBT 13 +

    Install Apache Zookeeper

     Download Apache Zookeeper from Apache Zookeeper site

     root@monitoringserver /]# wget http://www.us.apache.org/dist/zookeeper/current/apache-zookeeper-3.5.5-bin.tar.gz
    --2019-07-10 20:52:48--  http://www.us.apache.org/dist/zookeeper/current/apache-zookeeper-3.5.5-bin.tar.gz
    Resolving www.us.apache.org (www.us.apache.org)...
    Connecting to www.us.apache.org (www.us.apache.org)||:80... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 10622522 (10M) [application/x-gzip]
    Saving to: ‘apache-zookeeper-3.5.5-bin.tar.gz’

    100%[======================================================================================>] 10,622,522  11.6MB/s   in 0.9s

    2019-07-10 20:52:50 (11.6 MB/s) - ‘apache-zookeeper-3.5.5-bin.tar.gz’ saved [10622522/10622522]

    Untar zookeeper

    [root@monitoringserver /]# tar -xzf apache-zookeeper-3.5.5-bin.tar.gz

    Create Zookeeper Log Folder and Set Permission

    [root@monitoringserver /]# cd apache-zookeeper-3.5.5-bin
    [root@monitoringserver apache-zookeeper-3.5.5-bin]# ls
    bin  conf  docs  lib  LICENSE.txt  NOTICE.txt  README.md  README_packaging.txt
    [root@monitoringserver apache-zookeeper-3.5.5-bin]#

    [root@monitoringserver conf]# mkdir -p /zookeeper/data/logs
    [root@monitoringserver conf]# chmod -R 777 /zookeeper/
    [root@monitoringserver conf]#

    Change the directory to conf and Create a new file zoo.cfg.

    [root@monitoringserver conf]# vi zoo.cfg
    [root@monitoringserver conf]#

    Add following Content to zoo.cfg

    [root@monitoringserver conf]# cat zoo.cfg
    # The number of milliseconds of each tick
    # The number of ticks that the initial
    # synchronization phase can take
    # The number of ticks that can pass between
    # sending a request and getting an acknowledgement
    # the directory where the snapshot is stored.
    # do not use /tmp for storage, /tmp here is just
    # example sakes.
    # the port at which the clients will connect
    # the maximum number of client connections.
    # increase this if you need to handle more clients
    # Be sure to read the maintenance section of the
    # administrator guide before turning on autopurge.
    # http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
    # The number of snapshots to retain in dataDir
    # Purge task interval in hours
    # Set to "0" to disable auto purge feature
    [root@monitoringserver conf]#

    Start Zookeeper - 

    [root@monitoringserver apache-zookeeper-3.5.5-bin]# nohup bin/zkServer.sh start &
    [1] 59786
    [root@monitoringserver apache-zookeeper-3.5.5-bin]#

    Install Kafka

    To install a standalone kafka please follow this link.

    Install of  Yahoo Kafka Manager:

    [root@monitoringserver kafkaManager]# wget https://github.com/yahoo/kafka-manager/archive/master.zip
    --2019-07-10 20:07:19--  https://github.com/yahoo/kafka-manager/archive/master.zip
    Resolving github.com (github.com)...
    Connecting to github.com (github.com)||:443... connected.
    HTTP request sent, awaiting response... 302 Found
    Location: https://codeload.github.com/yahoo/kafka-manager/zip/master [following]
    --2019-07-10 20:07:19--  https://codeload.github.com/yahoo/kafka-manager/zip/master
    Resolving codeload.github.com (codeload.github.com)...
    Connecting to codeload.github.com (codeload.github.com)||:443... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: unspecified [application/zip]
    Saving to: ‘master.zip’

        [   <=>                                                                                 ] 1,106,343   2.18MB/s   in 0.5s

    2019-07-10 20:07:20 (2.18 MB/s) - ‘master.zip’ saved [1106343]

    [root@monitoringserver kafkaManager]#

    Install java

    [root@monitoringserver kafkaManager]# yum install java-1.8.0-openjdk
    [root@monitoringserver kafkaManager]# yum install java-1.8.0-openjdk-devel

    Unzip kafka-manager:

    [root@monitoringserver kafkaManager]# unzip master.zip

    Build kafka-manager

    [root@monitoringserver kafka-manager-master]# ./sbt clean dist
    Downloading sbt launcher for 1.2.8:
      From  http://repo.scala-sbt.org/scalasbt/maven-releases/org/scala-sbt/sbt-launch/1.2.8/sbt-launch.jar
        To  /root/.sbt/launchers/1.2.8/sbt-launch.jar
    Getting org.scala-sbt sbt 1.2.8  (this may take some time)...
    downloading https://repo1.maven.org/maven2/org/scala-sbt/sbt/1.2.8/sbt-1.2.8.jar ...
            [SUCCESSFUL ] org.scala-sbt#sbt;1.2.8!sbt.jar (17ms)
    downloading https://repo1.maven.org/maven2/org/scala-lang/scala-library/2.12.7/scala-library-2.12.7.jar ...
            [SUCCESSFUL ] org.scala-lang#scala-library;2.12.7!scala-library.jar (238ms)
    downloading https://repo1.maven.org/maven2/org/scala-sbt/main_2.12/1.2.8/main_2.12-1.2.8.jar ...
            [SUCCESSFUL ] org.scala-sbt#main_2.12;1.2.8!main_2.12.jar (86ms)
    [info] Packaging /kafkaManager/kafka-manager-master/target/scala-2.12/kafka-manager_2.12- ...
    [info] Done packaging.
    [success] All package validations passed
    [info] Your package is ready in /kafkaManager/kafka-manager-master/target/universal/kafka-manager-
    [success] Total time: 41 s, completed Jul 10, 2019 8:24:54 PM
    [root@monitoringserver kafka-manager-master]#

    Copy package and Unzip:

    cd  target/universal/
    unzip kafka-manager-
    cd kafka-manager-

    Start the Service using below command

     bin/kafka-manager -Dkafka-manager.zkhosts=”localhost:2181″

    or You can edit the "conf/application.conf"   with zookeeper values "kafka-manager.zkhosts=". Then start the Kafka Manager Service.


    Kafka Manager GUI:

    Add Cluster:

    Check more Kafka Articles:

    Check the Enable JMX Polling button and click on save. Kafka manager will register the cluster and provide you all information about it.

    Broker Details:

    Create a Topic:

    Topic Summary:

    Topic Details:

    Delete a Topic:

    1 comment: