Additional Required Kafka Broker and Client Setup

Owned by SL Docs
Last updated: Jan 30, 2020
1 min read

In order to monitor your Kafka brokers, consumers, zookeepers, and producers via JMX, additional options should be passed to the Kafka JVMs via environment variables to open JMX ports for monitoring purposes. For example:

export KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.authenticate=false \
-Dcom.sun.management.jmxremote.ssl=false \
-Djava.net.preferIPv4Stack=true \
-Dcom.sun.management.jmxremote.local.only=false \
-Djava.rmi.server.hostname=<host IP address>"

export JMX_PORT=9999
kafka-server-start.sh ...

export JMX_PORT=9998
kafka-console-consumer.sh ...

Depending on your organization's security policies, it may also be necessary to open ports in your host's firewall.

In order to monitor your topics and partitions and display consumer lag metrics, each consumer must have its client.id set to the same value as the name of the JMX connection to the consumer. For example, if you are using kafka-console-consumer and the property looks like this:

collector.sl.rtview.cache.config=kafka_consumer_cache_source.rtv $kafkaConn:consumer1 ...

... then the console-consumer command would include the following argument:

--consumer-property client.id=consumer1