Continental Innovates with Rancher and Kubernetes
This diagram shows how data flows through the Monitoring V2 application:
After collecting metrics from exporters, Prometheus stores the time series in a local on-disk time series database. Prometheus optionally integrates with remote systems, but rancher-monitoring uses local storage for the time series database.
The database can then be queried using PromQL, the query language for Prometheus. Grafana dashboards use PromQL queries to generate data visualizations.
The PromQL query language is the primary tool to query Prometheus for time series data.
In Grafana, you can right-click a CPU utilization and click Inspect. This opens a panel that shows the raw query results.The raw results demonstrate how each dashboard is powered by PromQL queries.
Rules define the conditions for Prometheus to fire alerts. When PrometheusRule custom resources are created or updated, the Prometheus Operator observes the change and calls the Prometheus API to synchronize the rule configuration with the Alerting Rules and Recording Rules in Prometheus.
When you define a Rule (which is declared within a RuleGroup in a PrometheusRule resource), the spec of the Rule itself contains labels that are used by Alertmanager to figure out which Route should receive this Alert. For example, an Alert with the label team: front-end will be sent to all Routes that match on that label.
A PrometheusRule allows you to define one or more RuleGroups. Each RuleGroup consists of a set of Rule objects that can each represent either an alerting or a recording rule with the following fields:
Prometheus doesn’t maintain the state of whether alerts are active. It fires alerts repetitively at every evaluation interval, relying on Alertmanager to group and filter the alerts into meaningful notifications.
The evaluation_interval constant defines how often Prometheus evaluates its alerting rules against the time series database. Similar to the scrape_interval, the evaluation_interval also defaults to one minute.
The rules are contained in a set of rule files. Rule files include both alerting rules and recording rules, but only alerting rules result in alerts being fired after their evaluation.
For recording rules, Prometheus runs a query, then stores it as a time series. This synthetic time series is useful for storing the results of an expensive or time-consuming query so that it can be queried more quickly in the future.
Alerting rules are more commonly used. Whenever an alerting rule evaluates to a positive number, Prometheus fires an alert.
The Rule file adds labels and annotations to alerts before firing them, depending on the use case:
The Alertmanager handles alerts sent by client applications such as the Prometheus server. It takes care of the following tasks:
Alertmanager coordinates where alerts are sent. It allows you to group alerts based on labels and fire them based on whether certain labels are matched. One top-level route accepts all alerts. From there, Alertmanager continues routing alerts to receivers based on whether they match the conditions of the next route.
While the Rancher UI forms only allow editing a routing tree that is two levels deep, you can configure more deeply nested routing structures by editing the Alertmanager custom resource YAML.
By editing the forms in the Rancher UI, you can set up a Receiver resource with all the information Alertmanager needs to send alerts to your notification system.
By editing custom YAML in the Alertmanager or Receiver configuration, you can also send alerts to multiple notification systems. For more information, see the section on configuring Receivers.
Prometheus Operator introduces a set of Custom Resource Definitions that allow users to deploy and manage Prometheus and Alertmanager instances by creating and modifying those custom resources on a cluster.
Prometheus Operator will automatically update your Prometheus configuration based on the live state of the resources and configuration options that are edited in the Rancher UI.
By default, a set of resources curated by the kube-prometheus project are deployed onto your cluster as part of installing the Rancher Monitoring Application to set up a basic Monitoring/Alerting stack.
The resources that get deployed onto your cluster to support this solution can be found in the rancher-monitoring Helm chart, which closely tracks the upstream kube-prometheus-stack Helm chart maintained by the Prometheus community with certain changes tracked in the CHANGELOG.md.
There are also certain special types of ConfigMaps and Secrets such as those corresponding to Grafana Dashboards, Grafana Datasources, and Alertmanager Configs that will automatically update your Prometheus configuration via sidecar proxies that observe the live state of those resources within your cluster.
PushProx enhances the security of the monitoring application, allowing it to be installed on hardened Kubernetes clusters.
To expose Kubernetes metrics, PushProxes use a client proxy model to expose specific ports within default Kubernetes components. Node exporters expose metrics to PushProx through an outbound connection.
The proxy allows rancher-monitoring to scrape metrics from processes on the hostNetwork, such as the kube-api-server, without opening up node ports to inbound connections.
PushProx is a DaemonSet that listens for clients that seek to register. Once registered, it proxies scrape requests through the established connection. Then the client executes the request to etcd.
All of the default ServiceMonitors, such as rancher-monitoring-kube-controller-manager, are configured to hit the metrics endpoint of the client using this proxy.
For more details about how PushProx works, refer to Scraping Metrics with PushProx.
rancher-monitoring deploys two exporters to expose metrics to prometheus: node-exporter and windows-exporter. Both are deployed as DaemonSets.
node-exporter exports container, pod and node metrics for CPU and memory from each Linux node. windows-exporter does the same, but for Windows nodes.
For more information on node-exporter, refer to the upstream documentation.
kube-state-metrics is also useful because it exports metrics for Kubernetes components.
When the monitoring application is installed, you will be able to edit the following components in the Rancher UI:
ServiceMonitors define targets that are intended for Prometheus to scrape. The Prometheus custom resource tells Prometheus which ServiceMonitors it should use to find out where to scrape metrics from.
The Prometheus Operator observes the ServiceMonitors. When it observes that ServiceMonitors are created or updated, it calls the Prometheus API to update the scrape configuration in the Prometheus custom resource and keep it in sync with the scrape configuration in the ServiceMonitors. This scrape configuration tells Prometheus which endpoints to scrape metrics from and how it will label the metrics from those endpoints.
Prometheus scrapes all of the metrics defined in its scrape configuration at every scrape_interval, which is one minute by default.
The scrape configuration can be viewed as part of the Prometheus custom resource that is exposed in the Rancher UI.
The Prometheus Deployment or StatefulSet scrapes metrics, and the configuration of Prometheus is controlled by the Prometheus custom resources. The Prometheus Operator watches for Prometheus and Alertmanager resources, and when they are created, the Prometheus Operator creates a Deployment or StatefulSet for Prometheus or Alertmanager with the user-defined configuration.
When the Prometheus Operator observes ServiceMonitors, PodMonitors and PrometheusRules being created, it knows that the scrape configuration needs to be updated in Prometheus. It updates Prometheus by first updating the configuration and rules files in the volumes of Prometheus’s Deployment or StatefulSet. Then it calls the Prometheus API to sync the new configuration, resulting in the Prometheus Deployment or StatefulSet to be modified in place.
Prometheus scrapes metrics from deployments known as exporters, which export the time series data in a format that Prometheus can ingest. In Prometheus, time series consist of streams of timestamped values belonging to the same metric and the same set of labeled dimensions.
To allow monitoring to be installed on hardened Kubernetes clusters, rancher-monitoring application proxies the communication between Prometheus and the exporter through PushProx for some Kubernetes master components.
The Kubernetes components that directly expose metrics to Prometheus are the following:
* For RKE and RKE2 clusters, ingress-nginx is deployed by default and treated as an internal Kubernetes component.
The purpose of this architecture is to allow us to scrape internal Kubernetes components without exposing those ports to inbound requests. As a result, Prometheus can scrape metrics across a network boundary.
The Kubernetes components that expose metrics to Prometheus through PushProx are the following:
For each PushProx exporter, we deploy one PushProx client onto all target nodes. For example, a PushProx client is deployed onto all controlplane nodes for kube-controller-manager, all etcd nodes for kube-etcd, and all nodes for kubelet. We deploy exactly one PushProx proxy per exporter.
The process for exporting metrics is as follows:
Metrics are scraped differently based on the Kubernetes distribution. For help with terminology, see Terminology(#terminology). For details, see the table below: