Prometheus is a free software application used for event monitoring and alerting. It records real-time metrics in a time series database built using an HTTP pull model, with flexible queries and real-time alerting. Prometheus is an open-source monitoring and alerting toolkit for containers and microservices. Based on the organizations that have adopted it, Prometheus has become the mainstream, open-source monitoring tool of choice for those that lean heavily on containers and microservices.

 Architecture of Prometheus

The Prometheus server works on the principle of scraping, i.e., invoking the metrics endpoints of the various nodes that it is configured to monitor. It collects these metrics at regular intervals and stores them locally. The nodes expose these over the endpoints that the Prometheus server scrapes.

An Exporter is a piece of software that gets existing metrics from a third-party system and exports them to the metric format that the Prometheus server can understand. A sample metric from a Prometheus server could be the current usage of free memory or file-system free via a Node Exporter in the Prometheus server.

The Prometheus server: This is the heart of the system. This server collects the metrics from multiple nodes and stores them locally. The Prometheus server works on the principle of scraping, i.e., invoking the metrics endpoints of the various nodes that it is configured to monitor. It collects these metrics at regular intervals and stores them locally. These metrics are pulled from nodes that run specific exporters (which are modules that extract information and translate it into the Prometheus format, which the server can then ingest). The nodes expose these over the endpoints that the Prometheus server scrapes.

Push gateway: In case the nodes are not exposing an endpoint from which the Prometheus server can collect the metrics, the Prometheus ecosystem has a push gateway. This gateway API is useful for one-off jobs that run, capture the data, transform that data into the Prometheus data format and then push that data into the Prometheus server.

Alert manager: One half of the Prometheus system is about collecting metrics. But of more importance is the ability to define your own alerts on those metrics, so that you can be notified in case of any discrepancies or levels that you might be interested in. This is the job of the alerts manager, which stores not just the alert levels but also can deliver these alerts to you over multiple channels like SMS, email, Slack, etc.

Visualization: Prometheus comes with its own user interface that you can use to check on the configuration, nodes, and graphs. Additionally, it is now compatible with Grafana, a leading open-source visualization application, so that Prometheus data is available for viewing inside Grafana. Prometheus also exposes an API, so in case you are interested in writing your own clients, you can do that too.