Illumina Innovates with Rancher and Kubernetes
Database workloads that require fast recovery can't afford manual intervention. StorageOS engineers will highlight challenges running databases in StatefulSets and demonstrate how to solve them on Rancher.
In this session of the Kubernetes Master Class, you'll learn how to think about observability on Kubernetes, how to use that to troubleshoot problems, and how this applies to various tools including Datadog.
Container monitoring environments come in all shapes and sizes. Some are open source while others are commercial. Some are available in the Rancher Catalog while others require manual configuration. Some are general purpose while others are aimed specifically at container environments. Some are hosted in the cloud while others require installation on own cluster hosts.
In this post, we take an updated look at 10 container monitoring solutions. This effort builds on earlier work including Ismail Usman’s Comparing 7 Monitoring Options for Docker from 2015 and The Great Container Monitoring Bake Off Meetup in October of 2016.
Since its founding in 2015, the Cloud Native Computing Foundation (CNCF) has become one of the most important movers and shakers in the open source ecosystem—especially when it comes to tools that affect containers and other “cloud-native” technologies. CNCF was established to promote and organize projects related to large-scale industry trends towards containerization, orchestration, and microservices architectures. In the time since, 10 open source projects have been added to the foundation.
In the previous part of this series, we have seen how to deploy an Elasticsearch Cluster using Rancher Catalog. Now it’s time to make good use of this catalog, right?
Introduction As a reminder, Elasticsearch is the cornerstone of the ELK platform (ELK stands for Elasticsearch/Logstash/Kibana). In this article, we’ll deploy the stack using Rancher Catalog, and use it to track tags and brands on Twitter. Tracking hashtags on Twitter can be very useful for measuring the impact of a Twitter-based marketing campaign.
Note: you can read the Part 1 and Part 2 of this series, which describes how to deploy service stacks from a private docker registry with Rancher. This is my third and final blog post, and follows part 2, where I stepped through the creation of a private, password-protected Docker registry. and integrated this private registry with Rancher. In this post, we will be putting this registry to work (although for speed, I will use public images).
 *by Stefan Thies (@seti321), DevOps evangelist at Sematext. * [The Rancher Community Catalog just got two new gems - SPM and Logsene - monitoring and logging tools from ]Sematext[. If you are familiar with Logstash, Kibana, Prometheus, Grafana, and friends, this post explains what SPM and Logsene bring to the Rancher users’ table, and how they are different from other monitoring or logging solutions.]
Meet Sematext Docker Agent [Sematext Docker Agent] is a modern, Docker-native monitoring and log collection agent.
Monitoring your container-based infrastructure is crucial to ensure good performance, identify issues early and gain the insight necessary to maximize its efficiency. When you are dealing with a large number of often short-lived containers spread over multiple hosts and even data centers, understanding the operational health of your infrastructure implies the need to aggregate performance data from both physical hosts as well as the container cluster running on top of it.
Elasticsearch is a Lucene-based search engine developed by the open-source vendor, elastic. With principal features like scalability, resiliency, and top-notch performance, it has overtaken Apache Solr, one of its closest competitors. Nowadays, Elasticsearch is almost everywhere where a search engine is involved: it’s the E of the well-known ELK stack, which makes it straightforward for your project to process analytics (the L stands for Logstash which is used to process data like logs, streams, metrics; K stands for Kibana, a data visualization platform – projects also managed by elastic).
Elasticsearch is one of the most popular analytics platform for large datasets. It is useful for a range of use-cases ranger from log aggregation, business intelligence as well as machine learning. Elasticsearch is popular because of its simple REST based API which makes it trivial to create indices, add data and make complex queries. However, before you get up and running building your dataset and running queries you need to setup a elasticsearch cluster, which can be a somewhat daunting prospect.
Once any application, dockerized or otherwise, reaches production, log aggregation becomes one of the biggest concerns. We will be looking at a number of solutions for gathering and parsing application logs from docker containers running on multiple hosts. This will include using a third-party service such as Loggly for getting setup quickly as well as bringing up an ELK stack (Elastic Search, Log Stash, Kibana) stack. We will look at using middleware such as FluentD to gather logs from Docker containers which can then be routed to one of the hundreds of consumers supported by fluentd.
Container logging is a common challenge for container deployments. Logging with containers is a bit different than traditional logging, because the logs for each container are nested within the container. On September 16th, we hosted an online meetup discussing all aspects of container logging, and demonstrating how to build a scalable logging service for Docker and Rancher that uses Elasticsearch, Logstash, and Kibana (ELK), along with Logspout. In the meetup Rancher DevOps lead Bill Maxwell discussed: • Docker Logging Challenges • Options for gathering logs from containers • System and Application logging requirements • Deploying an ELK stack using Docker Compose with Rancher • Scaling and managing a production ELK deployment You can view a recording of the meetup below.
At Rancher Labs we generate a lot of logs in our internal environments. As we conduct more and more testing on these environments we have found the need to centrally aggregate the logs from each environment. We decided to use Rancherto build and run a scalable ELK stack to manage all of these logs. For those that are unfamiliar with the ELK stack, it is made up of Elasticsearch, Logstash and Kibana.
I recently compared several docker monitoring tools and services. Since the article went live we have gotten feedback about additional tools that should be included in our survey. I would like to highlight two such tools; Prometheus and Sysdig cloud. Prometheus is a capable self-hosted solution which is easier to manage than sensu. Sysdig cloud on the other hand provides us with another hosted service much like Scout and Datadog.
In this article, Rancher compares seven Docker monitoring options and goes over some of the common tools used to monitor containers. Visit us to learn more.
Hi, I’m James Harris, (@sir_yogi_bear) one of the engineers here @Rancher_Labs, and I am excited to announce we added support this week for pulling and viewing Docker logs in Rancher. The addition of the feature allows users to easily work with their containers from the web UI in a much more involved way. Previously, there was no way to track the output of a container through Rancher. Now you can easily follow both the Std out and Std error of a container.