Continental Innovates with Rancher and Kubernetes
The Alertmanager Config Secret contains the configuration of an Alertmanager instance that sends out notifications based on alerts it receives from Prometheus.
By default, Rancher Monitoring deploys a single Alertmanager onto a cluster that uses a default Alertmanager Config Secret. As part of the chart deployment options, you can opt to increase the number of replicas of the Alertmanager deployed onto your cluster that can all be managed using the same underlying Alertmanager Config Secret.
This Secret should be updated or modified any time you want to:
By default, you can either choose to supply an existing Alertmanager Config Secret (i.e. any Secret in the cattle-monitoring-system namespace) or allow Rancher Monitoring to deploy a default Alertmanager Config Secret onto your cluster. By default, the Alertmanager Config Secret created by Rancher will never be modified / deleted on an upgrade / uninstall of the rancher-monitoring chart to prevent users from losing or overwriting their alerting configuration when executing operations on the chart.
cattle-monitoring-system
rancher-monitoring
For more information on what fields can be specified in this secret, please look at the Prometheus Alertmanager docs.
The full spec for the Alertmanager configuration file and what it takes in can be found here.
For more information, refer to the official Prometheus documentation about configuring routes.
When you define a Rule (which is declared within a RuleGroup in a PrometheusRule resource), the spec of the Rule itself contains labels that are used by Prometheus to figure out which Route should receive this Alert. For example, an Alert with the label team: front-end will be sent to all Routes that match on that label.
team: front-end
Available as of v2.5.4
Prerequisites: The monitoring application needs to be installed. If you configured monitoring with an existing Alertmanager Secret, it must have a format that is supported by Rancher’s UI. Otherwise you will only be able to make changes based on modifying the Alertmanager Secret directly. Note: We are continuing to make enhancements to what kinds of Alertmanager Configurations we can support using the Routes and Receivers UI, so please file an issue if you have a request for a feature enhancement.
Prerequisites:
To create notification receivers in the Rancher UI,
Result: Alerts can be configured to send notifications to the receiver(s).
The notification integrations are configured with the receiver, which is explained in the Prometheus documentation.
receiver
Rancher v2.5.4 introduced the capability to configure receivers by filling out forms in the Rancher UI.
The following types of receivers can be configured in the Rancher UI:
The custom receiver option can be used to configure any receiver in YAML that cannot be configured by filling out the other forms in the Rancher UI.
#<channelname>
SMTP options:
smtp.email.com
Events API v2
Prometheus
Opsgenie Responders:
The YAML provided here will be directly appended to your receiver within the Alertmanager Config Secret.
The Alertmanager must be configured in YAML, as shown in this example.
The route needs to refer to a receiver that has already been configured.
[ group_by: '[' <labelname>, ... ']' ]
cluster=A
alertname=LatencyHigh
'...'
group_by: ['...']
...
The Match field refers to a set of equality matchers used to identify which alerts to send to a given Route based on labels defined on that alert. When you add key-value pairs to the Rancher UI, they correspond to the YAML in this format:
match: [ <labelname>: <labelvalue>, ... ]
The Match Regex field refers to a set of regex-matchers used to identify which alerts to send to a given Route based on labels defined on that alert. When you add key-value pairs in the Rancher UI, they correspond to the YAML in this format:
match_re: [ <labelname>: <regex>, ... ]
To set up notifications via Slack, the following Alertmanager Config YAML can be placed into the alertmanager.yaml key of the Alertmanager Config Secret, where the api_url should be updated to use your Webhook URL from Slack:
alertmanager.yaml
api_url
route: group_by: ['job'] group_wait: 30s group_interval: 5m repeat_interval: 3h receiver: 'slack-notifications' receivers: - name: 'slack-notifications' slack_configs: - send_resolved: true text: '{{ template "slack.rancher.text" . }}' api_url: <user-provided slack webhook url here> templates: - /etc/alertmanager/config/*.tmpl
While configuring the routes for rancher-cis-benchmark alerts, you can specify the matching using the key-value pair job: rancher-cis-scan.
rancher-cis-benchmark
job: rancher-cis-scan
For example, the following example route configuration could be used with a Slack receiver named test-cis:
test-cis
spec: receiver: test-cis group_by: # - string group_wait: 30s group_interval: 30s repeat_interval: 30s match: job: rancher-cis-scan # key: string match_re: {} # key: string
For more information on enabling alerting for rancher-cis-benchmark, see this section.