Container monitoring environments come in all shapes and sizes. Some are open source while others are commercial. Some are available in the Rancher Catalog while others require manual configuration. Some are general purpose while others are aimed specifically at container environments. Some are hosted in the cloud while others require installation on own cluster hosts. In this post, we take an updated look at 10 container monitoring solutions. This effort builds on earlier work including Ismail Usman’s Comparing 7 Monitoring Options for Docker from 2015 and The Great Container Monitoring Bake Off Meetup in October of 2016.
Since its founding in 2015, the Cloud Native Computing Foundation (CNCF) has become one of the most important movers and shakers in the open source ecosystem—especially when it comes to tools that affect containers and other “cloud-native” technologies. CNCF was established to promote and organize projects related to large-scale industry trends towards containerization, orchestration, and microservices architectures. In the time since, 10 open source projects have been added to the foundation.
In the previous part of this series, we have seen how to deploy an Elasticsearch Cluster using Rancher Catalog. Now it’s time to make good use of this catalog, right? Introduction As a reminder, Elasticsearch is the cornerstone of the ELK platform (ELK stands for Elasticsearch/Logstash/Kibana). In this article, we’ll deploy the stack using Rancher Catalog, and use it to track tags and brands on Twitter. Tracking hashtags on Twitter can be very useful for measuring the impact of a Twitter-based marketing campaign.
Note: you can read the Part 1 and Part 2 of this series, which describes how to deploy service stacks from a private docker registry with Rancher. This is my third and final blog post, and follows part 2, where I stepped through the creation of a private, password-protected Docker registry. and integrated this private registry with Rancher. In this post, we will be putting this registry to work (although for speed, I will use public images).
 *by Stefan Thies (@seti321), DevOps evangelist at Sematext. * [The Rancher Community Catalog just got two new gems - SPM and Logsene - monitoring and logging tools from ]Sematext[. If you are familiar with Logstash, Kibana, Prometheus, Grafana, and friends, this post explains what SPM and Logsene bring to the Rancher users’ table, and how they are different from other monitoring or logging solutions.] Meet Sematext Docker Agent [Sematext Docker Agent] is a modern, Docker-native monitoring and log collection agent.
Monitoring your container-based infrastructure is crucial to ensure good performance, identify issues early and gain the insight necessary to maximize its efficiency. When you are dealing with a large number of often short-lived containers spread over multiple hosts and even data centers, understanding the operational health of your infrastructure implies the need to aggregate performance data from both physical hosts as well as the container cluster running on top of it.
Elasticsearch is a Lucene-based search engine developed by the open-source vendor, elastic. With principal features like scalability, resiliency, and top-notch performance, it has overtaken Apache Solr, one of its closest competitors. Nowadays, Elasticsearch is almost everywhere where a search engine is involved: it’s the E of the well-known ELK stack, which makes it straightforward for your project to process analytics (the L stands for Logstash which is used to process data like logs, streams, metrics; K stands for Kibana, a data visualization platform – projects also managed by elastic).
Elasticsearch is one of the most popular analytics platform for large datasets. It is useful for a range of use-cases ranger from log aggregation, business intelligence as well as machine learning. Elasticsearch is popular because of its simple REST based API which makes it trivial to create indices, add data and make complex queries. However, before you get up and running building your dataset and running queries you need to setup a elasticsearch cluster, which can be a somewhat daunting prospect.
Once any application, dockerized or otherwise, reaches production, log aggregation becomes one of the biggest concerns. We will be looking at a number of solutions for gathering and parsing application logs from docker containers running on multiple hosts. This will include using a third-party service such as Loggly for getting setup quickly as well as bringing up an ELK stack (Elastic Search, Log Stash, Kibana) stack. We will look at using middleware such as FluentD to gather logs from Docker containers which can then be routed to one of the hundreds of consumers supported by fluentd.
Container logging is a common challenge for container deployments. Logging with containers is a bit different than traditional logging, because the logs for each container are nested within the container. On September 16th, we hosted an online meetup discussing all aspects of container logging, and demonstrating how to build a scalable logging service for Docker and Rancher that uses Elasticsearch, Logstash, and Kibana (ELK), along with Logspout. In the meetup Rancher DevOps lead Bill Maxwell discussed: • Docker Logging Challenges • Options for gathering logs from containers • System and Application logging requirements • Deploying an ELK stack using Docker Compose with Rancher • Scaling and managing a production ELK deployment You can view a recording of the meetup below.