Continental Innovates with Rancher and Kubernetes
The Monitoring app sets prometheus.prometheusSpec.ignoreNamespaceSelectors=false, which enables monitoring across all namespaces by default.
prometheus.prometheusSpec.ignoreNamespaceSelectors=false
This ensures you can view traffic, metrics and graphs for resources deployed in a namespace with istio-injection=enabled label.
istio-injection=enabled
If you would like to limit Prometheus to specific namespaces, set prometheus.prometheusSpec.ignoreNamespaceSelectors=true. Once you do this, you will need to add additional configuration to continue to monitor your resources.
prometheus.prometheusSpec.ignoreNamespaceSelectors=true
This limits monitoring to specific namespaces.
Result: Prometheus will be limited to specific namespaces which means one of the following configurations will need to be set up to continue to view data in various dashboards
There are two different ways to enable Prometheus to detect resources in other namespaces when prometheus.prometheusSpec.ignoreNamespaceSelectors=true:
additionalScrapeConfig
This option allows you to define which specific services or pods you would like monitored in a specific namespace.
The usability tradeoff is that you have to create the service monitor or pod monitor per namespace since you cannot monitor across namespaces.
Prerequisite: Define a ServiceMonitor or PodMonitor for <your namespace>. An example ServiceMonitor is provided below.
<your namespace>
kubectl create -f <name of service/pod monitor file>.yaml
cat<< EOF | kubectl apply -f -
EOF
kubectl label namespace <your namespace> istio-injection=enabled
Result: <your namespace> can be scraped by prometheus.
apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: envoy-stats-monitor namespace: istio-system labels: monitoring: istio-proxies spec: selector: matchExpressions: - {key: istio-prometheus-ignore, operator: DoesNotExist} namespaceSelector: any: true jobLabel: envoy-stats endpoints: - path: /stats/prometheus targetPort: 15090 interval: 15s relabelings: - sourceLabels: [__meta_kubernetes_pod_container_port_name] action: keep regex: '.*-envoy-prom' - action: labeldrop regex: "__meta_kubernetes_pod_label_(.+)" - sourceLabels: [__meta_kubernetes_namespace] action: replace targetLabel: namespace - sourceLabels: [__meta_kubernetes_pod_name] action: replace targetLabel: pod_name
This enables monitoring across namespaces by giving Prometheus additional scrape configurations.
The usability tradeoff is that all of Prometheus’ additionalScrapeConfigs are maintained in a single Secret. This could make upgrading difficult if monitoring is already deployed with additionalScrapeConfigs prior to installing Istio.
additionalScrapeConfigs
prometheus.prometheusSpec.additionalScrapeConfigs
Result: All namespaces with the istio-injection=enabled label will be scraped by prometheus.
- job_name: 'istio/envoy-stats' scrape_interval: 15s metrics_path: /stats/prometheus kubernetes_sd_configs: - role: pod relabel_configs: - source_labels: [__meta_kubernetes_pod_container_port_name] action: keep regex: '.*-envoy-prom' - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port] action: replace regex: ([^:]+)(?::\d+)?;(\d+) replacement: $1:15090 target_label: __address__ - action: labelmap regex: __meta_kubernetes_pod_label_(.+) - source_labels: [__meta_kubernetes_namespace] action: replace target_label: namespace - source_labels: [__meta_kubernetes_pod_name] action: replace target_label: pod_name