This section describes the required configurations as well as the optional configurations available for the Solution Package for Apache Kafka in the RTView Configuration Application. You must define the classpath to the Apache Kafka jar files and you must also define data source connections for each connection that you want to monitor.
Note: In order to monitor your Kafka brokers, consumers, zookeepers, and producers via JMX, additional options should be passed to the JVMs. See Additional Required Kafka Broker and Client Setup for more information.
Note: See Creating Secure Connections for additional information on creating secure connections for Apache Kafka.
The CONNECTIONS tab displays.
Note: No setup is required for Solution Package Configuration > RTView Manager or Data Cache.
The Add Connection dialog displays.
Broker Connection
Connection Name: The name of the connection/server.
Connection Type: Select Broker from this drop down list.
Host: The IP address of the host.
JMX Port: The JMX port used when connecting.
Username: The username is used when creating the connection. This field is optional (if no username/password is set for the broker).
Password: This password is used when creating the connection. This field is optional (if no username/password is set for the broker). By default, the password entered is hidden. Click the associated "eye" icon to view the password text.
Cluster Name: The name of the cluster.
Topic Names: Optionally enter one or more topics to restrict monitoring to only those topics on the broker (all topics are monitored by default). You can enter multiple topics by adding a comma or clicking the Tab key after each. Once entered, you can click the X next to the topic name to remove them.
Client Port: Enter the Client Port to use this broker to monitor topic partitions for this cluster.
Consumer Connection
Connection Name: The name of the connection/server.
Connection Type: Select Consumer from this drop down list.
Host: The IP address of the host.
JMX Port: The JMX port used when connecting.
Username: The username is used when creating the connection. This field is optional (if no username/password is set for the consumer).
Password: This password is used when creating the connection. This field is optional (if no username/password is set for the consumer). By default, the password entered is hidden. Click the associated "eye" icon to view the password text.
Cluster Name: The name of the cluster.
Producer Connection
Connection Name: The name of the connection/server.
Connection Type: Select Producer from this drop down list.
Host: The IP address of the host.
JMX Port: The JMX port used when connecting.
Username: The username is used when creating the connection. This field is optional (if no username/password is set for the producer).
Password: This password is used when creating the connection. This field is optional (if no username/password is set for the producer). By default, the password entered is hidden. Click the associated "eye" icon to view the password text.
Cluster Name: The name of the cluster.
Zookeeper Connection
Connection Name: The name of the connection/server.
Connection Type: Select Zookeeper from this drop down list.
Client Port: Optionally enter the Client Port if you want to monitor topic partitions for this cluster.
Host: The IP address of the host.
JMX Port: The JMX port used when connecting.
Username: The username is used when creating the connection. This field is optional (if no username/password is set for the zookeeper).
Password: This password is used when creating the connection. This field is optional (if no username/password is set for the zookeeper). By default, the password entered is hidden. Click the associated "eye" icon to view the password text.
Cluster Name: The name of the cluster.
You can specify the number of history rows to store in memory, the compaction rules, the duration before metrics are expired and deleted, and the different types of metrics that you want the Historian to store in the DATA STORAGE tab in the RTView Configuration Application.
You can modify the maximum number of history rows to store in memory in the DATA STORAGE tab. The History Rows property defines the maximum number of rows to store for the KafkaZookeeper, KafkaServer, KafkaConsumer, KafkaProducer, KafkaServerMetrics, KafkaServerHistogram, KafkaServerTimer, KafkaServerTopicMetrics, KafkaTopicPartition, KafkaTopicTotalsByTopic, KafkaTopicTotalsByServer, KafkaTopicTotalsByConsumer, KafkaTopicTotalsByTopicAndServer, and KafkaTopicTotalsByTopicAndConsumer caches. The default settings for History Rows is 50,000. To update the default settings:
Data compaction, essentially, is taking large quantities of data and condensing it using a defined rule so that you store a reasonably sized sample of data instead of all of your data, thus preventing you from potentially overloading your database. The available fields are:
Condense Interval – The time interval at which the cache history is condensed for the following caches: KafkaZookeeper, KafkaServer, KafkaConsumer, KafkaProducer, KafkaServerMetrics, KafkaServerHistogram, KafkaServerTimer, KafkaServerTopicMetrics, KafkaTopicPartition, KafkaTopicTotalsByTopic, KafkaTopicTotalsByServer, KafkaTopicTotalsByConsumer, KafkaTopicTotalsByTopicAndServer, and KafkaTopicTotalsByTopicAndConsumer. The default is 60 seconds.
Condense Raw Time -- The time span of raw data kept in the cache history table for the following caches: KafkaZookeeper, KafkaServer, KafkaConsumer, KafkaProducer, KafkaServerMetrics, KafkaServerHistogram, KafkaServerTimer, KafkaServerTopicMetrics, KafkaTopicPartition, KafkaTopicTotalsByTopic, KafkaTopicTotalsByServer, KafkaTopicTotalsByConsumer, KafkaTopicTotalsByTopicAndServer, and KafkaTopicTotalsByTopicAndConsumer. The default is 1200 seconds.
History Time Span -- The duration of time to retain a row of cached data based on its timestamp.
The cache trims its History table by removing rows with timestamps that are older than the limit specified here. Specify the duration in seconds or specify a number followed by a single character indicating the desired time interval (for example, 15m for 15 minutes). The format is a number followed by one of the following valid characters:
y - years (365 days)
M - months (31 days)
w - weeks (7 days)
d - days
h - hours
m - minutes
s - seconds
Example: 1M
Note that this setting only determines the duration of rows kept in the History table by the cache data source. It does not affect database storage, if any, associated with the cache.
The caches impacted by this field are: KafkaZookeeper, KafkaServer, KafkaRegisteredBrokers, KafkaConsumer, KafkaProducer, KafkaServerMetrics, KafkaServerHistogram, KafkaServerTimer, KafkaServerTopicMetrics, KafkaTopicPartition, KafkaTopicTotalsByTopic, KafkaTopicTotalsByServer, KafkaTopicTotalsByConsumer, KafkaTopicTotalsByTopicAndServer, and KafkaTopicTotalsByTopicAndConsumer
Compaction Rules: This field defines the rules used to condense your historical data in the database for the following caches: KafkaZookeeper, KafkaServer, KafkaConsumer, KafkaProducer, KafkaServerMetrics, KafkaServerHistogram, KafkaServerTimer, KafkaServerTopicMetrics, KafkaTopicPartition, KafkaTopicTotalsByTopic, KafkaTopicTotalsByServer, KafkaTopicTotalsByConsumer, KafkaTopicTotalsByTopicAndServer, and KafkaTopicTotalsByTopicAndConsumer. By default, the columns kept in history will be aggregated by averaging rows with the following rule 1h -;1d 5m;2w 15m, which means the data from 1 hour will not be aggregated (1h - rule), the data over a period of 1 day will be aggregated every 5 minutes (1d 5m rule), and the data over a period of 2 weeks old will be aggregated every 15 minutes (2w 15m rule).
To update these settings:
Note: When you click in the Compaction Rules field, the Copy default text to clipboard link appears, which allows you copy the default text (that appears in the field) and paste it into the field. This allows you to easily edit the string rather than creating the string from scratch.
The data for each metric is stored in a specific cache and, when the data is not updated in a certain period of time, that data will either be marked as expired or, if it has been an extended period of time, it will be deleted from the cache altogether. By default, metric data will be set to expired when the data in the cache has not been updated within 45 seconds. Also, by default, if the data has not been updated in the cache within 3600 seconds, it will be removed from the cache.
The caches impacted by the Expire Time and Delete Time properties are: KafkaZookeeper, KafkaServer, KafkaRegisteredBrokers, KafkaConsumer, KafkaProducer, KafkaServerMetrics, KafkaServerHistogram, KafkaServerTimer, KafkaServerTopicMetrics, KafkaTopicPartition, KafkaTopicTotalsByTopic, KafkaTopicTotalsByServer, KafkaTopicTotalsByConsumer, KafkaTopicTotalsByTopicAndServer, and KafkaTopicTotalsByTopicAndConsumer. To modify these defaults:
The History Storage section allows you to select which metrics you want the Historian to store in the history database. By default, all historical data (in the KafkaServer, KafkaServerTopicMetrics, KafkaServerMetrics, KafkaServerHistogram, KafkaServerTimer, KafkaZookeeper, KafkaConsumer, KafkaProducer, KafkaTopicPartition, KafkaTopicTotalsByTopic, KafkaTopicTotalsByServer, KafkaTopicTotalsByConsumer, KafkaTopicTotalsByTopicAndServer, and KafkaTopicTotalsByTopicAndConsumer caches) is saved to the database. To disable the collection of historical data, perform the following steps:
The History Table Name Prefix field allows you to define a prefix that will be added to the database table names so that the Monitor can differentiate history data between data servers when you have multiple data servers with corresponding Historians using the same solution package(s) and database. In this case, each Historian needs to save to a different table, otherwise the corresponding data server will load metrics from both Historians on startup. Once you have defined the History Table Name Prefix, you will need to create the corresponding tables in your database as follows:
Note: If you are using Oracle for your Historian Database, you must limit the History Table Name Prefix to 2 characters because Oracle does not allow table names greater than 30 characters (and the longest table name for the solution package is 28 characters).
To add the prefix: