Continental Innovates with Rancher and Kubernetes
In Rancher 2.5, the monitoring application was improved. There are now two ways to enable monitoring and alerting. The older way is documented in this section, and the new application for monitoring and alerting is documented here.
To keep your clusters and applications healthy and driving your organizational productivity forward, you need to stay informed of events occurring in your clusters and projects, both planned and unplanned. When an event occurs, your alert is triggered, and you are sent a notification. You can then, if necessary, follow up with corrective actions.
This section covers the following topics:
Notifiers and alerts are built on top of the Prometheus Alertmanager. Leveraging these tools, Rancher can notify cluster owners and project owners of events they need to address.
Before you can receive alerts, you must configure one or more notifier in Rancher.
When you create a cluster, some alert rules are predefined. You can receive these alerts if you configure a notifier for them.
For details about what triggers the predefined alerts, refer to the documentation on default alerts.
Some examples of alert events are:
When you edit an alert rule, you will have the opportunity to configure the alert to be triggered based on a Prometheus expression. For examples of expressions, refer to this page.
Monitoring must be enabled before you can trigger alerts with custom Prometheus queries or expressions.
You can set an urgency level for each alert. This urgency appears in the notification you receive, helping you to prioritize your response actions. For example, if you have an alert configured to inform you of a routine deployment, no action is required. These alerts can be assigned a low priority level. However, if a deployment fails, it can critically impact your organization, and you need to react quickly. Assign these alerts a high priority level.
The scope for alerts can be set at either the cluster level or project level.
At the cluster level, Rancher monitors components in your Kubernetes cluster, and sends you alerts related to:
After you set up cluster alerts, you can manage each alert object. To manage alerts, browse to the cluster containing the alerts, and then select Tools > Alerts that you want to manage. You can:
As a cluster owner, you can configure Rancher to send you alerts for cluster events.
Prerequisite: Before you can receive cluster alerts, you must add a notifier.
Finally, choose the notifiers to send the alerts to.
Click Create.
Result: Your alert is configured. A notification is sent when the alert is triggered.
This alert type monitor for events that affect one of the Kubernetes master components, regardless of the node it occurs on.
Each of the below sections corresponds to a part of the alert rule configuration section in the Rancher UI.
Select the System Services option, and then select an option from the dropdown:
The alert will be triggered when the selected Kubernetes master component is unhealthy.
Select the urgency level of the alert. The options are:
Select the urgency level based on the importance of the service and how many nodes fill the role within your cluster. For example, if you’re making an alert for the etcd service, select Critical. If you’re making an alert for redundant schedulers, Warning is more appropriate.
etcd
By default, the below options will apply to all alert rules within the group. You can disable these advanced options when configuring a specific rule.
This alert type monitors for specific events that are thrown from a resource type.
Choose the type of resource event that triggers an alert. The options are:
Select a resource type from the Choose a Resource drop-down that you want to trigger an alert.
Select the urgency level of the alert.
Select the urgency level of the alert by considering factors such as how often the event occurs or its importance. For example:
This alert type monitors for events that occur on a specific node.
Select the Node option, and then make a selection from the Choose a Node drop-down.
Choose an event to trigger the alert.
Select the urgency level of the alert based on its impact on operations. For example, an alert triggered when a node’s CPU raises above 60% deems an urgency of Info, but a node that is Not Ready deems an urgency of Critical.
This alert type monitors for events that occur on any node on marked with a label. For more information, see the Kubernetes documentation for Labels.
Select the Node Selector option, and then click Add Selector to enter a key value pair for a label. This label should be applied to one or more of your nodes. Add as many selectors as you’d like.
Available as of v2.4.0
This alert type is triggered based on the results of a CIS scan.
Select CIS Scan.
Choose an event to trigger the alert:
This alert type monitors for the overload from Prometheus expression querying, it would be available after you enable monitoring.
Input or select an Expression, the dropdown shows the original metrics from Prometheus, including:
Choose a comparison:
If applicable, choose a comparison value or a threshold for the alert to be triggered.
Select a duration for a trigger alert when the expression value crosses the threshold longer than the configured duration.
Select the urgency level of the alert based on its impact on operations. For example, an alert triggered when a node’s load expression sum(node_load5) / count(node_cpu_seconds_total{mode="system"}) raises above 0.6 deems an urgency of Info, but 1 deems an urgency of Critical.
sum(node_load5) / count(node_cpu_seconds_total{mode="system"})