DataServer and DataCollector Architecture for RTView Enterprise Monitor

Owned by SL Docs
Last updated: Jan 17, 2020
12 min read

Overview

 

The above diagram shows how data flows through the RTViewDataServer and RTViewDataCollector deployments. The Data Server connects to the Monitored Components to collect metric data that it stores in local caches and uses to generate alerts. In cases where the data collection needs to be distributed, one or more Data Collectors can be deployed to connect to the monitored components and forward the collected data to the Data Server. The Historian is an optional process that stores historical metric and alert data to the RTVHISTORY database. When the Historian is enabled, the Data Server will query historical data from the RTVHISTORY on startup to populate in-memory history and also any time the Enterprise Monitor request history data that is older than the data in the in-memory history. The Configuration Application is a browser based application for configuring the RTView processes. It connects to the Data Server and Data Collector to read and write Configuration Files. 

The RTViewDataServer does not contain a user interface. It needs to be integrated with Enterprise Monitor in order to view metric and alert data from the Data Server and also allow the user to enable, disable, and set thresholds on alerts. 

The following sections provide a more detailed description of each process.

Data Server

The Data Server gathers and caches the data from the applications being monitored and also hosts the alerts for that data. Because the Data Server can exist behind firewalls, it simplifies and strengthens the secured delivery of information to clients beyond the firewall. The Data Server serves metric and alert data to the Enterprise Monitor Central Server via socket or rtvdata servlet on port 3278 and receives data via socket from the optional Data Collector on port 3272. It also serves metrics and alert data to the Enterprise Monitor HTML UI via the rtvquery servlet that connects via socket on port 3278. The Historian runs in the same directory as the Data Server, while the optional Data Collector(s) typically run in a different directory or a different system. By default, socket connections to the Data Server are unsecured. The Data Server supports secure socket connections (SSL) with or without certificates. It also supports client whitelist and blacklist. See the Data Server Security section in the RTView Core User Guide for more information on secure socket and client whitelist/blacklist configurations.

When integrated with Enterprise Monitor, the RTViewDataServer will connect to the Enterprise Monitor Central Server via socket on port 10070 or via the rtview-central-rtvdata servlet as configured on the EM Integration tab of the Configuration Application. By default, socket and servlet connections to the Central Server are unsecured. See the Enterprise Monitor Central Server section for information on securing this connection. 

The Enterprise Monitor will make 2 connections to the RTViewDataServer:

  1. The Central Server and Central Display Server will connect to this Data Server via socket on port 3278 either directly or via rtvdata servlet. By default, the socket is used but you can modify the connection to use the servlet in the EM Integration tab in the Configuration Application. While the rtvdata servlet cannot be configured for authentication, Tomcat access filters can be used to restrict access. The rtvdata servlet will connect via secure socket if the Data Server is configured for SSL sockets.

  2. The Central Server HTML UI will connect to the Data Server via the rtvquery servlet that connects to the Data Server via socket on port 3278. By default, it connects to the unsecured rtvquery servlet that is hosted in Jetty. This means that all Enterprise Monitor browser clients that use the HTML UI need permission to access port 3270 on the host running the RTViewDataServer. If this is not desirable, or if you want to enable authentication in the rtvquery servlet, configure and run update_wars in the projects/rtview-server directory and deploy the war files to the Tomcat installation included with the Central Server. With this deployment, only the system where Enterprise Monitor is installed will need permission to access the host running the RTViewDataServer. After deploying your servlets to Tomcat, modify the connections in the EM Integration tab in the Configuration Application to point to the servlets in Tomcat. NOTE: Once you have enabled authentication in the servlet, you must use Tomcat or another application server as it will no longer work in Jetty. It is recommended that you deploy your authenticated servlets to the Tomcat installation in RTViewCentral as this will allow single sign on between the HTML UI and the rtvquery servlet. If you deploy your secured rtvquery servlet to an application server other than the one in RTView Central, you will need to modify the rtview-central.war file as follows to configure credentials:

    1. For each SP rtvquery servlet that is secured and is hosted on a different tomcat than the UI servlet, an entry must be added to the data server array in setup.js, as follows:

      1. in a command prompt or terminal window, cd to projects/rtview-server

      2. extract setup.js from rtview-central.war: jar -xf rtview-central.war setup.js

      3. add an entry to the data server array

         ,{
            name: 'dataserver_name',
            typeC: 'Data',
            serverAddressC: 'dataserver_rtvuqery_url',
            authValueC: 'Basic ' + btoa('rtvuser:rtvuser')
         }

        ... where dataserver_name is the name of the SP data server as it appears in the Name column of the Central server's RtvSPDataServerConns cache, dataserver_rtvuqery_url is the url of that data server's rtvquery servlet as it appears in QueryURL column, and the second rtvuser is the password for the rtvuser role.

      4. pack the modified setup.js back into rtview-central.war: jar -uf rtview-central.war setup.js

The Data Collector is an optional process that is used to distribute connections to Monitored Components Data Collectors instead of having the Data Server connect to each component to be monitored directly. This process collects data from Monitored Components and forwards it to the Data Server via socket or the rtvagent servlet. See the Data Collector section below for information on securing this connection.

The Configuration Application connects to the Data Server via the rtvadmin servlet to read and write properties files. The rtvadmin servlet connects to the Data Server via socket on port 3278. See the Configuration Application section below for information about servlet authentication. The rtvadmin servlet will connect via secure socket if the Data Server is configured for SSL sockets.

If the Historian is enabled, the Data Server connects to the RTVHISTORY database on startup to read initial cache history data and if Enterprise Monitor displays request history data older than the in-memory cache history.

The Data Server opens a jmx port on 3268 to enable monitoring. By default, the jmx port is not secured, but can be secured via SSL and username/password.

The Data Server runs an internal Jetty application server that hosts its servlets on port 3270. It also hosts a simple cache viewer at http://localhost:3270/common. See the Application Servers section for information on disabling or securing Jetty.

Alerts

When integrated with Enterprise Monitor, enabling and setting alert thresholds is done via the Enterprise Monitor user interface and stored in the Enterprise Monitor's ALERTDEFS database. The RTViewDataServer accesses alert thresholds via its connection the Enterprise Monitor CentralServer. Optional alert notifications are configured and executed in the Enterprise Monitor. Since the notification actions are user defined, security must be determined by the user.

Monitored Components

Monitored components consist of EMS Servers, Hawk MicroAgents, BW Servers, Solace Brokers, and so on. See Creating Secure Connections for more information on securing connections to the monitored components.

Data Collector

This process is optional and is used to distribute connections to Monitored Components Data Collectors instead of having the Data Server connect to each component to be monitored directly. This process collects data from Monitored Components and forwards it to the Data Server via socket on port 3272 or the rtvagent servlet. In the RTView Configuration Application > Data Server > COLLECTOR tab, the Target definition determines whether data is sent to a socket or a servlet. If the URL for the target is host:port, then it will be sent via socket, which is not secured by default. This socket can be secured via SSL by specifying the following property on the CUSTOM PROPERTIES tab in the Configuration Application of each receiver Data Server:

Property Name: sl.rtview.rtvagent.ssl

Property Value: true

Property Fitler: collector

If the url is the receiver's rtvagent servlet, it will send data to that rtvagent servlet, which will connect via socket to the Data Server on port 3272 that can be secured via SSL as described above. While the rtvagent servlet cannot be configured for authentication, Tomcat access filters can be used to restrict access and it can be deployed on https. While the Data Collector typically does not have data clients, it accepts data requests via socket on port 3276 that can be secured as described in the Data Server section.The Data Collector also opens jmx on port 3266 for monitoring. By default, the jmx port is not secured, but can be secured via SSL and username/password.

Configuration Application

The Configuration Application connects to the Data Server via the rtvadmin servlet, which is configured with http authentication. It should be run on https since user credentials are not encrypted. Passwords saved by the configuration application are scrambled except in the case where they are added in the CUSTOM PROPERTIES section. See the Data Server section above for information about securing the connection between the Configuration Application and Data Server. 

Configuration Files

Configuration (.properties) files are stored on the file system and read by all RTView processes to control configuration. Additionally, the Configuration Application writes these files, scrambling all connection and database passwords. Passwords entered in the CUSTOM PROPERTIES tab are not scrambled. 

Historian

The Historian connects to the Data Server via socket and saves cache history to a database via jdbc. This process is optional and the user can configure which data will be saved. See the Data Server section above for information about securing the connection between the Historian and Data Server. See the Database section below for information about the connection between the Historian and the database. This process opens jmx port 3267 for monitoring. By default, the jmx port is not secured. See the Security section below for details on securing the Historian.

Database

When integrated with Enterprise Monitor, the local ALERTDEFS database is not used. The RTVHISTORY database stores cache data (if the Historian is enabled). The Historian connects to the RTVHISTORY database to insert cache history data and to perform data compaction. The Data Server connects to the RTVHISTORY database on startup to load initial history into the caches and also when the user interface asks for history data older than what is contained in the in-memory history caches.  

By default, the Data Server and Historian will connect to the HSQLDB database that is included with RTView using an unsecured jdbc connection. See the Hsqldb documentation for information on configuring it for secure jdbc connections. Alternately, you can use your own database and secure the jdbc connection according to the documentation for that database. 

Application Servers

By default, the Data Server runs a Jetty process that hosts all of the RTView servets and accepts http client requests on port 3270. You can optionally configure Jetty to use https instead of http. This will require you to provide a certificate for your domain. Once you have a certificate, do the following in the Configuration Application in the Data Server tab:

  • Turn on the Use Https toggle.

  • Set the Keystore File field to the keystore file name (including the path) that contains the certificate for your domain.

  • Optionally enter the Keystore Password and Key Manager Password if they are required for your keystore.

  • Save your configuration and restart the data server.

The Configuration Application and HTML UI use http authentication and require the following roles that are preconfigured. You can modify the user names and passwords (but not the roles) in RTVAPM_HOME/common/lib/ext/jetty/rtvadmin-users.xml:

  • rtvadmin

  • rtvuser

  • rtvalertmgr

Jetty does not limit the number of failed login attempts, which leaves it open to brute force attacks. If this is a concern, you should deploy with Tomcat or another Application Server.

You can optionally use Tomcat or another Application Server in addition to or instead of the Jetty process that comes with RTView. To deploy your servlets to your application server, go into the projects/rtview-server directory and run update_wars.bat or update_wars.sh. Copy all of the generated war files to the webapps directory in your application server. 

Tomcat and most other Application Servers can be configured for https. This will require you  to provide a certificate for your domain. Follow the application server documentation for enable https. Additionally, Tomcat access filters can be configured to restrict access according to the remote client host or address. Tomcat also has a feature named LockOut Realm to protect against brute force login attacks. After five successive login attempts for a given username with invalid password, then all logins for that username are rejected for the next five minutes. The LockOut Realm parameters are configurable. See the Tomcat documentation for more information.

You will need to add the following roles to your Application Server for use with the Configuration Application and HTML UI authentication. For Tomcat, users and roles are defined in conf\tomcat-users.xml:

  • rtvadmin

  • rtvuser

  • rtvalertmgr

You can optionally disable Jetty in the Data Server when using Tomcat or another Application Server. To disable Jetty, you must access the Configuration Application from Tomcat or another Application Server. In the Configuration Application, go to the Data Server tab and do the following:

  • Turn off the HTML Server Enabled toggle.

  • Save your configuration and restart. 

Security

Security options per RTView process are included in the section for each component above. This section provides a summary of security options for the entire deployment organized by priority. 

  1. Secure Installation Location - High Priority. The RTView installation and Application Server should be run in a secure location to ensure displays and configuration files are secure and access-restricted.

  2. Login and Servlet Authentication - High Priority. 

    1. The HTML UI connects to the Data Server via the rtvquery servlet. The rtvquery servlet does not have authentication enabled by default. See the Data Server section in this document for information on enabling authentication in the rtvquery servlet. 

    2. Configuration Application - By default, the Configuration Application is configured with http authentication which should use https since http authentication does not encrypt user credentials. 

  3. Application Server Security - High Priority

    1. It is highly recommended that you configure your Application Server to use https as described in the Application Server section of this document. The RTView servlets that support http authentication do not encrypt user credentials. 

    2. It is highly recommended that you change the user credentials in your Application Server for the rtvadmin, rtvuser and rtvalertmgr roles since the default credentials are documented and publicly available.  

  4. Secure Connections between RTView Processes - Medium-to-Low Priority*

    1. The Historian, Data Server, Data Collector, rtvquery servlet, rtvdata servlet, rtvadmin servlet and rtvagent servlet all connect to the Data Server via socket which is unsecured by default. The Data Server supports secure socket connections (SSL) with or without certificates. It also supports client whitelist and blacklist. See the Data Server Security section in the RTView Core User Guide for more information on secure socket and client whitelist/blacklist configurations.

  5. Secure Connections to Monitored Components - Medium-to-Low Priority*

    1. The Data Server uses component specific api's to connect to Monitored Components. Securing these connections is described here.

  6. Secure Connections to Databases - Medium-to-Low Priority*

    1. The Data Server and Historian both create database connections using JDBC. See the Database section in this document for information on securing jdbc connections to your database.

  • If Secured Installation Location has been met, these are lower priority.

Securing JMX

The Data Server, Data Collector, Historian and Display Server all open JMX ports for monitoring. By default, these jmx ports are unsecured, but they can be secured either by user name and password or by SSL. . To configure the JMX ports of the RTView processes to use SSL and/or username and password, go to the SECURITY tab in the RTView Configuration Application. Note that the start/stop/status_rtv scripts use JMX to communicate with the RTView processes. If you secure the JMX ports on the RTView processes, you will need to save arguments for use by the scripts in the Configuration Application SECURITY tab or pass additional arguments into the start/stop/status_rtv scripts.

To secure with user name and password, you will need a java password file. To secure with SSL, you will need a certificate, truststore and keystore. These are described in the Oracle Java documentation.

Generating SSL Certificates, Truststores and Keystores

When the Data Server socket is secured via SSL with certificate, an SSL handshake is performed between the client and the server when the client attempts to connect. For this handshake, the client must provide a certificate the server trusts, and the server must provide a certificate the client trusts. A keystore file contains the application's certificate and private key. A truststore file contains the application's trusted certificates. Since this is a two way handshake, the client and the Data Server will both need a keystore and a truststore. These files are created by running the Java keytool on the command line. The following steps generate self-signed certificates for host1(server) and host2(client) along with client and server side truststore (client_truststore.jks and server_truststore.jks) and keystore (client_keystore.jks and server_keystore.jks) files. Typically certificates would be signed by a trusted certificate authority. The file names and passwords used here are arbitrary.

On the command line, execute the following:

keytool -genkey -keystore server_keystore.jks -alias rtview_server -validity 3650 -keyalg RSA -storepass mypassword -keypass mypassword -dname cn=host1
keytool -list -v -keystore server_keystore.jks -storepass mypassword
keytool -export -alias rtview_server -keystore server_keystore.jks -storepass mypassword -file server.cer
keytool -importcert -alias rtview_server -file server.cer -v -noprompt -keystore client_truststore.jks -storepass mypassword
keytool -genkey -keystore client_keystore.jks -alias rtview_client -validity 3650 -keyalg RSA -storepass mypassword -keypass mypassword -dname cn=host2
keytool -export -alias rtview_client -keystore client_keystore.jks -storepass mypassword -file client.cer
keytool -importcert -alias rtview_client -file client.cer -v -noprompt -keystore server_truststore.jks -storepass mypassword

For more information on how to use keytool, see the Oracle Java documentation.

For additional information on securing connections to your monitored components, see Creating Secure Connections. To see additional information on the Security tab in the Configuration Application, see Security Tab.