[Sematext Docker
Agent] *by Stefan Thies (@seti321), DevOps evangelist at Sematext. * [The Rancher Community Catalog just got two new gems - SPM and Logsene - monitoring and logging tools from ]Sematext[. If you are familiar with Logstash, Kibana, Prometheus, Grafana, and friends, this post explains what SPM and Logsene bring to the Rancher users’ table, and how they are different from other monitoring or logging solutions.]

Meet Sematext Docker Agent

[Sematext Docker Agent] is a modern, Docker-native monitoring and log collection agent. It runs as a tiny container on every Docker host, and collects logs, metrics, and events for all cluster nodes and their containers. The agent discovers all containers on all nodes managed by Rancher. After the deployment of Sematext Docker Agent, all logs, Docker events, and metrics are immediately available out of the box. Why is this valuable? It means you don’t have to spend the next N hours or days figuring out which data to collect, or how to chart it. Plus, you don’t need to resources to maintain your own logging and monitoring infrastructure - your Docker metrics and events end up getting shipped to SPM and logs get shipped to Logsene (SPM is an application performance monitoring service supporting a number of integrations, including Docker, while Logsene is a log management service - a hosted ELK stack that works well with both Kibana and Grafana). Sematext metrics
stack

DevOps Tools Comparison

There are several open source tools for Docker logs and metrics, such as cAdvisor and Logspout. Unfortunately, none of these is very comprehensive, so one is often forced to combine a number of these tools, which leads down a Franken-monitoring path. This later becomes a costly technical debt that no one enjoys fixing. Thus, while one might think Sematext Docker Agent is not much more than a combination of cAdvisor and Logspout, especially in regards to log management, Sematext Docker Agent has features such as format detection, log parsing, data enrichment (Geo-IP, tagging with metadata), and log routing.

Configure Docker Agent in Rancher Catalog

To setup Sematext Docker Agent with Rancher, you can configure it with the appropriate catalog template (search for \“Sematext\” in the community catalog). Let’s have a quick look at a few key Sematext Docker Agent features, and how to configure them with Rancher.

Automatic Log Tagging for Docker Compose and Kubernetes

All logs are automatically tagged with metadata. This includes support for Docker/Rancher Compose projects and Kubernetes.

For Docker Containers

  • Container ID
  • Container name
  • Image name

For Docker/Rancher Compose:

  • Service Name
  • Project Name
  • Container number (if you use scale=N)

For Kubernetes:

Note that Kubernetes containers’ logs are not much different from those of Docker container logs. However, Kubernetes users need to access logs for deployed pods, so it’s very useful to capture Kubernetes-specific information available for log search, such as:

  • Name space
  • Pod name
  • Container name
  • Kubernetes UID

TIP: to enable Kubernetes tagging, set [Kubernetes=1]

Log Routing

For large deployments, you may want to index logs for different tenants or applications in separate indices/Logsene apps (this also lets you separate who can view which logs). We’ve made this easy: simply tag your containers with a Docker label, or set the LOGSENE_TOKEN environment variable (LOGSENE_TOKEN=your app token) and Sematext Docker Agent will ship logs to the correct index! This way you don’t need any central configuration file to map containers and indices/tokens, and log routing becomes very dynamic and flexible.

Integrated Log Parser

[Handling of logs is based on the Docker API and a library called ]logagent-js[, open-sourced by Sematext. This parser framework includes patterns for log format detection and parsing of a number of different log formats used by official Docker images such as: ]

  • [Web servers like Nginx, Apache httpd or anything using common web server log format]
  • [Search engines like Elasticsearch and Solr]
  • [Message queues like Apache Kafaka and nsq.io ]
  • [Databases like MongoDB, HBase, MySQL]
  • [Detection of JSON log format, often used by Node.js applications e.g., bunyan and winston logging frameworks]

TIP: to create custom patterns add them to the [LOGAGENT_PATTERNS] catalog template field.

Automatic Geo-IP Enrichment for Container Logs

[Getting logs from Docker Containers ]collected[, ]shipped[ and ]parsed[ out of the box is already a big time saver, but some application logs need additional ]enrichment[ with information from other data sources. A common use case is to enrich web server logs (or really, any logs with IP addresses) with geographical information derived from those IP addresses. ] [Sematext Docker Agent supports ]Geo-IP enrichment of docker logs[. It uses Maxmind Geo-IP lite database, which it periodically updates automatically for you. There is no need to stop the container or mount new volumes with the Geo-IP database, or any other manual work.]

TIP: to enable Geo-IP enrichment set the environment variable [GEOIP_ENABLED=true].

Filter Container Logs

[In some cases it makes sense to collect only logs from your critical applications and skip less critical and noisy services (e.g. frequently running cleanup jobs). To do that you can whitelist and blacklist containers by image name or container name. The settings take regular expressions, which are matched against the relevant metadata fields.]

Whitelist containers

[MATCH_BY_NAME]

[Regular expression to white list container names]

[MATCH_BY_IMAGE]

[Regular expression to white list image names]

Blacklist containers

[SKIP_BY_NAME]

[Regular expression to black list container names]

[SKIP_BY_IMAGE]

[Regular expression to black list image names for logging]

How to use the Sematext Catalog Entry

When you run the Rancher server user interface, simply search in the community catalog for \“sematext\“, \“monitoring\” or \“logs\“. Rancher Catalog monitoring
search [Choose “View Details”, and in the “Configuration Options” enter the SPM and Logsene App tokens. You can obtain these from ][https://apps.sematext.com,][ where you can sign up and create your SPM and Logsene apps. If your Rancher cluster runs behind a firewall, you might need to specify the proxy URL in the HTTPS_PROXY or HTTP_PROXY environment variable. If you run Kubernetes on the same cluster, choose KUBERNETES=1. ] [If you’d like to collect all logs, just press “Launch” without specifying any filter for containers or images. ] Sematext catalog
detail Sematext catalog
options Sematext
stack

Summary

[We hope this introduction to Sematext Docker Agent in Rancher Catalog helps you get started with Docker monitoring and logging, without having to take the dreaded Franken-monitoring path. A complete list of configuration parameters is available on ]Github[. We think the new catalog template covers the most relevant options, but if you see anything important missing feel free to contribute to the ]Rancher community catalog[ by submitting an issue or pull request. Give ]Sematext Docker Agent[ a try with Rancher and benefit from a managed service and free plans to watch your cattle while you sleep. Thanks to the great Rancher Community Catalog, it is super easy to get monitoring and logging set up and working in no time. ]