In this blog, we’ll discuss how and why Weave developed the best-practice RED method for monitoring apps with Prometheus.
What is Prometheus Monitoring?
You may have heard a lot about Prometheus lately, especially when it comes to monitoring applications in Kubernetes. To provide a bit of background before we delve into the RED method, apps running in containers and orchestrated by Kubernetes are highly automated and dynamic, and so, when it comes to monitoring applications in these environments, traditional server-based monitoring tools designed for static services are not sufficient.
This is where Prometheus comes in.
Prometheus is an open source project that was originally developed by engineers at SoundCloud. It was built and designed specially to monitor microservices that run in containers. Data is scraped from running services at time intervals and saved to a time-series database where it can be queried via the PromQL language. Because the data is stored as a time series, it allows you to explore those time intervals to diagnose problems when they occurred and to also analyze long-term monitoring trends with your infrastructure — two awesomely powerful features of Prometheus.
At Weaveworks we built on the open source distribution of Prometheus and created a scalable, multi-tenant version that is part of our Software-as-a-Service called Weave Cloud.
After having run this service for several months now, and by using Weave Cloud to monitor itself, we’ve learned a few things about monitoring cloud native applications and devised a system that we use in determining what to measure before instrumenting code.
What to Instrument?
One of the most important decisions to make when setting up Prometheus Monitoring is deciding on the type of metrics you need to collect about your app. The metrics you choose simplifies troubleshooting when a problem occurs and also enables you to stay on top of the stability of your services and infrastructure. To help us think about what’s important to instrument, we defined a system that we call the RED method. Read more
Prometheus is a modern and popular monitoring alerting system, built at SoundCloud and eventually open sourced in 2012 – it handles multi-dimensional time series data really well, and friends at InfinityWorks have already developed a Rancher template to deploy Prometheus at click of a button.
In hybrid cloud environments, it is likely that one might be using multiple orchestration engines such as Kubernetes and Mesos, in which case it is helpful to have the stack or application portable across environments. In this short tutorial, we will convert the template for Prometheus from Cattle format to make it work in a Kubernetes environment. It is assumed that the reader has a basic understanding of Kubernetes concepts such as pods, replication controller (RC), services and so on. If you need a refresher on the basic concepts, the Kubernetes 101 and concept guide are excellent starting points.
During the meetup Darren Shepherd demonstrated how to deploy a complete container stack
On July 15th, Darren Shepherd and Shannon Williams hosted an online meetup demonstrating how to deploy a pilot Docker Service, and teaching attendees how to implement an integrated stack that included DockerHub, GitHub, Rancher, Jenkins and Prometheus. We’ve recorded the meeting and shared it below. You can register for our next online meetup on our events page. If you would like to learn more about Rancher, please sign up for our Beta Program, or schedule a discussion with one of our engineers. Read more
I have already talked about several ways to monitor docker containers and also using Prometheus to monitor Rancher deployments. However, until now it has been a manual process of launching monitoring agents on our various hosts. With the release of the Rancher beta with scheduling and support for Docker compose we can begin to make monitoring a lot more automated. In today’s post we will look at using Rancher’s new “Rancher compose” tool to bring up our deployment with a single command, using scheduling to make sure we have a monitoring agent running on every host, and using labels to isolate and present our metrics. I will again be using Prometheus to present our container metrics and the Container-Exporter agent to capture metrics on our Docker hosts. If you have not already please do take a look at earlier articles (here and here) describing how to setup and use Prometheus. I am writing this article assuming you already have a Rancher environment stood up. Read more
I have blogged about monitoring docker deployments a couple times now (here & here), however, up to this point we have been monitoring container stats without looking at the bigger picture. How do these containers fit into a larger unit and how we get insights into the deployment as a whole rather than individual containers. In this post I will cover leveraging docker labels and Rancher’s projects and services support to provide monitoring information that understands the deployment structure. I will be using Prometheus as our monitoring system because in our earlier survey we found it to be the best self-hosted solution. However, you could use the same approach of inspecting labels to integrate with any monitoring system.
Before we start monitoring a deployment we have to bring up a deployment to monitor. For this purpose I will be creating a mock deployment with a Web Service, and a Database Service. The web service consists of nginx containers behind an load balancer and the database service will be a set of mysql containers. We will create two Projects; development and production, which both contain the web and database services . Lets get started by provisioning a Rancher management container. First, create the Rancher server using the command shown below. Note that we will be using a tagged version (0.21.5-rc1) of Rancher as the labels support is not yet available in the default version of Rancher. Keep in mind, Rancher is still in alpha and is changing quickly, this version has an updated UI, and a handful of new features. Read more
I recently compared several docker monitoring tools and services. Since the article went live we have gotten feedback about additional tools that should be included in our survey. I would like to highlight two such tools; Prometheus and Sysdig cloud. Prometheus is a capable self-hosted solution which is easier to manage than sensu. Sysdig cloud on the other hand provides us with another hosted service much like Scout and Datadog. Collectively they help us add more choices to their respective classes. As before I will be using the following six criteria to evaluate Prometheus and Sysdig cloud: 1) ease of deployment, 2) level of detail of information presented, 3) level of aggregation of information from entire deployment, 4) ability to raise alerts from the data and 5) Ability to monitor non-docker resources 6) cost. Read more