Moving Containers to Production – A Short Checklist

on Jan 10, 2017

containers to production checklistIf you’re anything like me, you’ve been watching the increasing growth of container-based solutions with considerable interest, and you’ve probably been experimenting with a couple of ideas. At some point in the future, perhaps you’d like to take those experiments and actually put them out there for people to use. Why wait? It’s a new year, and there is no time like the present to take some action on that goal.

Experimenting is great, and you learn a great deal, but often in the midst of trying out new things, hacking different technologies together and making it all work, things get introduced into our code which probably shouldn’t be put into a production environment. Sometimes, having a checklist to follow when we’re excited and nervous about deploying new applications out into the wild can help ensure that we don’t do things we shouldn’t. Consider this article as the start of a checklist to ready your Docker applications for prime time.

Item 1: Check Your Sources

Free eBook: Continuous Integration and Deployment with Docker and Rancher

Years ago, I worked on a software project with a fairly large team. We started running into a problem—Once a week, at 2 PM on a Tuesday afternoon, our build would start failing. At first we blamed the last guy to check his code in, but then it mysteriously started working before he could identify and check-in a fix. And then the next week it happened again. It took a little research, but we traced the source of the failure to a dependency in the project which had been set to always pull the latest snapshot release from the vendor, and it turned out that the vendor had a habit of releasing a new, albeit buggy version of their library around 2 PM on Tuesday afternoons.

Using the latest and greatest versions of a library or a base image can be fun in an experiment, but it’s risky when you’re relying on it in a production environment. Scan through your Docker configuration files, and check for two things.

First, ensure that you have your source images tied to a stable version of the image. Any occurrence of :latest in your Docker configuration files should fail the smell test.

Second, if you are using Dockerhub as your image repository, use the official image wherever possible. Among the reasons for doing this: “These repositories have clear documentation, promote best practices, and are designed for the most common use case.” (Official Repositories on Docker Hub)

Item 2: Keep your Secrets…Secret

As Gandalf asked, “Is it secret? Is it safe?” Our applications have a need for secret information. Most applications have a need for a combination of database credentials, API tokens, SSH keys and other necessary information which is not appropriate, or advisable for a public audience. Secret storage is one of the biggest weaknesses of container technology. Some solutions which have been implemented, but are not recommended are:

Baking the secrets into the image. Anyone with access to the registry can potentially access the secrets, and if you have to update them, this can be a rather tedious process.

Using volume mounts. Unfortunately, this keeps all of your secrets in a single and static location, and usually requires them to be stored in plain text.

Using environment variables. These are easily accessible by all processes using the image, and are usually easily viewed with Docker inspect.

Encrypted solutions. Secrets are stored in an encrypted state, with decryption keys on the host machines. While your passwords and other key data elements aren’t stored in plain text, they are fairly easy to locate, and the decryption methods identified.

The best solution at this point is to use a secrets store, such as Vault by HashiCorp or Keywhiz from Square. Implementation is typically API-based and very reliable. Once implemented, a secret store provides a virtual filesystem to an application, which it can use to access secured information. Each store provides documentation on how to set up, test and deploy a secret store for your application.

Item 3: Secure the Perimeter

A compelling reason for the adoption of a container-based solution is the ability to share resources on a host machine. What we gain in ease of access to the host machine’s resources, however, we lose in the ability to separate the processes from a single container from those of another. Great care needs to be taken to ensure that the user under which a containers application is started has the minimum required privileges on the underlying system.

In addition, it is important that we establish a secure platform on which to launch our containers. We must ensure that the environment is protected wherever possible from the threat of external influences. Admittedly this has less to do with the containers themselves, and more with the environment into which they are deployed, but it is important nonetheless.

Item 4: Make Sure to Keep an Eye on Things

The final item on this initial checklist for production-readying your application is to come up with a monitoring solution. Along with secret management, monitoring is an area related to container-based applications which is still actively evolving. When you’re experimenting with an application, you typically don’t run it under much significant load, or in a multiple-user environment. Additionally, for some reason, our users insist on finding new and innovative ways to leverage the solutions we provide, which is both a blessing and a curse.

This article “Comparing monitoring options for Docker deployments” provides information and comparison between a number of monitoring options, as does a more recent online meetup on the topic. The landscape for Docker monitoring solutions is still under continued development.

Go Forth and Containerize in an Informed Manner

The container revolution is without a doubt one of the most exciting and disruptive developments in the world of software development in recent years. Docker is the tool which all the cool kids are using, and let’s be honest, we all want to be part of that group. When you’re ready to take your project from an experimental phase into production, make sure you’re proceeding in an informed manner. The technology is rapidly evolving, and offers many advantages over traditional technologies, but be sure that you do your due diligence and confirm that you’re using the right tool for the right job.

 

Mike Mackrory is a Global citizen who has settled down in the Pacific Northwest – for now.  By day he works as a Senior Engineer on a Quality Engineering team and by night he writes, consults on several web based projects and runs a marginally successful eBay sticker business.  When he’s not tapping on the keys, he can be found hiking, fishing and exploring both the urban and the rural landscape with his kids.

Supercharge your CI/CD Pipeline with Containers

This hands-on guidebook provides a detailed overview of how to integrate containers into your build, test, and deployment processes to dramatically improve software delivery and operations.

Free eBook: CI/CD Pipelines with Docker and Rancher