Illumina Innovates with Rancher and Kubernetes
Introduction Jenkins has been the industry standard CI tool for years. It contains a multitude of functionalities, with almost 1,000 plugins in its ecosystem, this can be daunting to some who appreciate simplicity. Jenkins also came up in a world before containers, though it does fit nicely into the environment. This means that there is not a particular focus on the things that make containers great, though with the inclusion of Blue Ocean and pipelines, that is rapidly changing.
The cloud vs. on-premises debate is an old one. It goes back to the days when the cloud was new and people were trying to decide whether to keep workloads in on-premises datacenters or migrate to cloud hosts. But the Docker revolution has introduced a new dimension to the debate. As more and more organizations adopt containers, they are now asking themselves whether the best place to host containers is on-premises or in the cloud.
Docker has been a source of excitement and experimentation among developers since March 2013, when it was released into the world as an open source project. As the platform has become more stable and achieved increased acceptance from development teams, a conversation about when and how to move from experimentation to the introduction of containers into a continuous integration environment is inevitable. What form that conversation takes will depend on the players involved and the risk to the organization.
What do Docker containers have to do with Infrastructure as Code (IaC)? In a word, everything. Let me explain. When you compare monolithic applications to microservices, there are a number of trade-offs. On the one hand, moving from a monolithic model to a microservices model allows the processing to be separated into distinct units of work. This lets developers focus on a single function at a time, and facilitates testing and scalability.
Last week we announced a partnership with EVRY, one of the leading IT companies in the Nordics, to deliver Rancher’s container management platform as a service to EVRY customers. This is an exciting moment for Rancher as the service will introduce our software to a new audience looking to embrace DevOps and transform how they deliver IT. Not surprisingly, our relationship with EVRY began last year when a couple of their cloud architects downloaded Rancher and built a small test deployment.
One of the great things about microservices is that they allow engineering to decouple software development from application lifecycle. Every microservice:
can be written in its own language, be it Go, Java, or Python can be contained and isolated form others can be scaled horizontally across additional nodes and instances is owned by a single team, rather than being a shared responsibility among many teams communicates with other microservices through an API a message bus must support a common service level agreement to be consumed by other microservices, and conversely, to consume other microservices These are all very cool features, and most of them help to decouple various software dependencies from each other.
Prometheus is a modern and popular monitoring alerting system, built at SoundCloud and eventually open sourced in 2012 – it handles multi-dimensional time series data really well, and friends at InfinityWorks have already developed a Rancher template to deploy Prometheus at click of a button.
In hybrid cloud environments, it is likely that one might be using multiple orchestration engines such as Kubernetes and Mesos, in which case it is helpful to have the stack or application portable across environments.
The ultimate goal for a developer is to have their own micro data center, enabling them to test their services in an exact live replica. However, the life of a developer is full of compromises. Data is a reduced set or anonymized, and companies aren’t quite ready to pay for a data center per developmer. Today, i’ll provide an overview of how using Rancher and a local machine can eliminate some of these compromises.
Raul Sanchez is a microservices and Dev0ps architect in the innovation department at BBVA, exploring new technologies, bringing them to the company and the production lifecycle. In his spare time, he is a developer who collaborates on open source projects. He’s spent more than 20 years working on GNU/Linux and unix systems in different areas and sectors. Introduction GoCD is a Java open source continuous delivery system from ThoughtWorks.
John Patterson (@cantrobot) and Chris Lunsford run This End Out, an operations and infrastructure services company. You can find them online at https://www.thisendout.com *and follow them on twitter @thisendout. * Update: All four parts of the series are now live, you can find them here: Part 1: Getting started with CI/CD and Docker Part 2: Moving to Compose blueprints Part 3: Adding Rancher for OrchestrationPart 4: Completing the Cycle with Service Discovery This post is the first in a series in which we’d like to share the story of how we implemented a container deployment workflow using Docker, Docker-Compose and Rancher.
Today we achieved a major milestone by shipping Rancher 1.0, our first generally available release. After more than one and a half years of development, Rancher has reached the quality and feature completeness for production deployment. We first unveiled a preview of Rancher to the world at the November 2014 Amazon Re:invent conference. We followed that with a Beta release in June 2015. I’d like to congratulate the entire Rancher development team for this achievement.
Over the last few months our team at Rancher Labs has been adding support for Kubernetes within Rancher. We’ve been implementing Kubernetes in a way that takes advantage of Rancher’s platform orchestration, simple UI, access control, networking and storage capabilities to deliver simple to deploy Kubernetes clusters for managing applications. In our February meetup we introduced this new support, and discussed how these environments compare with our traditional Docker environments and help users understand when and how each can be used to deploy and manage container deployments.
So far in this series of articles we have looked at creating continuous integration pipelines using Jenkins and continuously deployingto integration environments. We also looked at using Rancher compose to run deployments as well as Route53 integration to do basic DNS management. Today we will cover production deployments strategies and also circle back to DNS management to cover how we can run multi-region and/or multi-data-center deployments with automatic fail-over. We also look at some rudimentary auto-scaling so that we can automatically respond to request surges and scale back when request rate drops again.
Docker Compose is a great framework for deploying application stacks, and at Rancher we’ve been working hard to make it possible to leverage that framework to create a catalog of application blueprints that can be repeatably configured and deployed. In this recording of our January online meetup, we demonstrated the new Catalog feature in Rancher and how to create catalog items. In the meetup we demonstrated: - Using the Rancher catalog to configure, deploy and upgrade an application - Creating a private app catalog linked to a git repo - Best practices for building catalog templates - Inserting application configuration into templates We demonstrated all of this live, and answered dozens of questions about Docker, Rancher, and building application templates.
Over the last year we have written about getting several application stacks running on top of docker, e.g. Magento, Jenkins, Prometheus and so forth. However, containerized deployment can be useful for more than just defining application stacks. In this series of articles we would like to cover an end-to-end development pipeline and discuss how to leverage Docker and Rancher in its’ various stages. Specifically, we’re going to cover; building code, running tests, packaging artifacts, continuous integration and deployment, as well as managing an application stack in production.
Meetup Screenshot: Bill Maxwell Demonstrates Sysdig monitoring his Rancher environment Yesterday we hosted an online meetup with the team from Sysdig, in which we discussed best practices for Docker monitoring, and some of the unique challenges around applying monitoring policies to containers. Over the course of the meetup, we introduced Rancher and Sysdig, and demonstrated how we’re using Sysdig here at Rancher to manage our containers. The meetup included a number of presentations, and we’ve included the agenda below along with direct links to that portion of the meetup if you’d like to jump ahead at all.
Rancher has come a long way since its early versions, and is becoming quite good at managing Docker applications and deploying complex services. That said, as your stacks move from simple demos to production applications you quickly realize that you need to know more about your Docker environment when you configure and upgrade container services. To address that problem, we recently introduced a container metadata service with Rancher v0.38.0, similar to Amazon’s Instance Metadata service.
Once any application, dockerized or otherwise, reaches production, log aggregation becomes one of the biggest concerns. We will be looking at a number of solutions for gathering and parsing application logs from docker containers running on multiple hosts. This will include using a third-party service such as Loggly for getting setup quickly as well as bringing up an ELK stack (Elastic Search, Log Stash, Kibana) stack. We will look at using middleware such as FluentD to gather logs from Docker containers which can then be routed to one of the hundreds of consumers supported by fluentd.
This year we sponsored the DockerConHackathon, and had an amazing 24 hours working with people hacking on Docker, Rancher, RancherOS and more. Two of our team, Darren Shepherd and Alena Prokharchyk were judges, so we didn’t think it would be fair to enter the contest. That said, we wanted to be involved in the hacking anyway so we built a little tool called SherDock, a simple image management tool for garbage collection, identifying orphaned volumes and more.
Today we announced the beta availability of Rancher, our open source Docker infrastructure and management software. It is an exciting day for our team, and a great opportunity to say thank you to all the people who have worked on the open source Rancher project, blogged and tweeted about using Rancher, and helped other new users on our support groups. For the last seven months, Rancher has been an alpha project, and throughout that time we’ve had a wonderful group of early users and testers trying out the product, suggesting enhancements, documenting bugs, and contributing code.
I have blogged about monitoring docker deployments a couple times now (here & here), however, up to this point we have been monitoring container stats without looking at the bigger picture. How do these containers fit into a larger unit and how we get insights into the deployment as a whole rather than individual containers. In this post I will cover leveraging docker labels and Rancher’s projects and services support to provide monitoring information that understands the deployment structure.
Since I started playing with Docker I have been thinking that its network implementation is something that will need to be improved before I could really use it in production. It is based on container links and service discovery but it only works for host-local containers. This creates issues for a few use cases, for example when you are setting up services that need advanced network features like broadcasting/multicasting for clustering.
As you may have seen, Rancher recently announced our integration with docker-machine. This integration will allow users to spin up Rancher compute nodes across multiple cloud providers right from the Rancher UI. In our initial release, we supported Digital Ocean. Amazon EC2 is soon to follow and we’ll continue to add more cloud providers as interest dictates. We believe this feature will really help the Zero-to-Docker _(and Zero-to-Rancher)_ experience. But the feature itself is not the focus of this post.
In the first part of this post, I created a full Node.js application stack using MongoDB as the application’s database and Nginx as a load balancer that distributed incoming requests to two Node.js application servers. I created the environment on Rancher and using Docker containers.
In this post I will go through setting up Rancher authentication with GitHub, and creating a webhook with GitHub for automatic deployments.
Rancher Access Control Starting from version 0.
So last week I finally got out from my “tech” comfort zone, and tried to set up a Node.js application which uses a MongoDB database, and to add an extra layer of fun I used Rancher to set up the whole application stack using Docker containers.
I designed a small application with Node, its only function is to calculate the number of hits on the website, you can find the code at Github