KAFKA CONNECT PROMETHEUS EXPORTER HOW TO

Recording rules allow you to precompute frequently needed or computationally expensive expressions and save their result as a new set of time series.

You can also use the configuration file to define recording rules and alerting rules: Prometheus uses a configuration file in YAML format to define the scraping jobs and their instances. Cluster health: Event Streams has a preconfigured monitoring dashboard, but other tools could be used as it exports a set of metrics via standard like JMX.Monitoring deployment health using User Interface, or OpenShift CLI.Here are a set of helpful links for Event Streams monitoring:

KAFKA CONNECT PROMETHEUS EXPORTER HOW TO

In the next section, we will demonstrate exactly that we will start with sample dashboards and make few modifications to exemplify how to configure key Kafka metrics to display in the dashboard. The Monitoring Kafka metrics article by DataDog and How to monitor Kafka by Server Density provides guidance on key Kafka and Prometheus metrics, reasoning to why you should care about them and suggestions on thresholds to trigger alerts. But you are still left to figure out which ones you want to actively monitor and the ones that you want to be actively alerted.Īn simple way to get started would be to start with the Grafana’s sample dashboards for the Prometheus exporters you chose to use and then modify them as you learn more about the available metrics and/or your environment. The easiest way to see the available metrics is to fire up jconsole and point it at a running kafka client or Kafka/Prometheus server this will allow browsing all metrics with JMX. What to monitor ¶Ī long list of metrics is made available by Kafka ( here) and Zookeeper ( here). Alternatively, you can consider writing your own custom exporter. Kafka exporter, Kafka Zookeeper Exporter by CloudFlare, and others). To monitor Kafka, for example, the JMX exporter is often used to provide broker level metrics, while community exporters claim to provide more accurate cluster level metrics (e.g. Some of them can be used in addition to the JMX export. There is also a number of exporters maintained by the community to explore. Kafka Broker, Zookeeper and Java clients (producer/consumer) expose metrics via JMX (Java Management Extensions) and can be configured to report stats back to Prometheus using the JMX exporter maintained by Prometheus. Producer(s) / Consumer(s), in general sense, which includes Kafka Connector cluster.ZooKeeper metrics as Kafka relies on it to maintain its state.Kafka Monitoring Kafka Monitoring Table of contentsĪ comprehensive Kafka monitoring plan should collect metrics from the following components:.

Kafka connect prometheus exporter