Tag: cicd

CICD Debates: Drone vs Jenkins

January 31, 2018

Introduction

Jenkins has been the industry standard CI tool for years. It contains a multitude of functionalities, with almost 1,000 plugins in its ecosystem, this can be daunting to some who appreciate simplicity. Jenkins also came up in a world before containers, though it does fit nicely into the environment. This means that there is not a particular focus on the things that make containers great, though with the inclusion of Blue Ocean and pipelines, that is rapidly changing.

Drone is an open source CI tool that wears simple like a badge of honor. It is truly Docker native; meaning that all actions take place within containers. This makes it a perfect fit for a platform like Kubernetes, where launching containers is an easy task.

Both of these tools walk hand in hand with Rancher, which makes standing up a robust Kubernetes cluster an automatic process. I’ve used Rancher 1.6 to deploy a K8s 1.8 cluster on GCE; as simple as can be.

This article will take Drone deployed on Kubernetes (on Rancher), and compare it to Jenkins across three categories:

  1. Platform installation and management
  2. Plugin ecosystem
  3. Pipeline details

In the end, I’ll stack them up side by side and try to give a recommendation. As usually is the case however, there may not be a clear winner. Each tool has its core focus, though by nature there will be overlap.

Prereqs

Before getting started, we need to do a bit of set up. This involves setting up Drone as an authorized Oauth2 app with a Github account. You can see the settings I’ve used here. All of this is contained within the Drone documentation.

There is one gotcha which I encountered setting up Drone. Drone maintains a passive relationship with the source control repository. In this case, this means that it sets up a webhook with Github for notification of events. The default behavior is to build on push and PR merge events. In order for Github to properly notify Drone, the server must be accessible to the world. With other, on-prem SCMs, this would not be the case, but for the example described here it is. I’ve set up my Rancher server on GCE, so that it is reachable from Github.com.

Drone installs from a container through a set of deployment files, just like any other Kubernetes app. I’ve adapted the deployment files found in this repo. Within the config map spec file, there are several values we need to change. Namely, we need to set the Github-related values to ones specific to our account. We’ll take the client secret and client key from the setup steps and place them into this file, as well as the username of the authorized user. Within the drone-secret file, we can place our Github password in the appropriate slot.

This is a major departure from the way Jenkins interacts with source code. In Jenkins, each job can define its relationship with source control independent of another job. This allows you to pull source from a variety of different repositories, including Github, Gitlab, svn, and others. As of now, Drone only supports git-based repos. A full list is available in the documentation, but all of the most popular choices for git-based development are supported.

We also can’t forget our Kubernetes cluster! Rancher makes it incredibly easy to launch and manage a cluster. I’ve chosen to use latest stable version of Rancher, 1.6. We could’ve used the new Rancher 2.0 tech preview, but constructing this guide worked best with the stable version. however, the information and steps to install should be the same, so if you’d like to try it out with newer Rancher, go ahead!

Task 1 – Installation and Management

Launching Drone on Kubernetes and Rancher is as simple as copy paste. I used the default K8s dashboard to launch the files. Uploading them one by one, starting with the namespace and config files, will get the ball rolling. [Here are some of the deployment files I used](https://github.com/appleboy/drone-on-kubernetes/tree/master/gke). I pulled from this repository and made my own local edits. This repo is owned by a frequent Drone contributor, and includes instructions on how to launch on GCE, as well as AWS. The Kubernetes yaml files are the only things we need here. To replicate, just edit the ConfigMap file with your specific values. Check out one of my files below.


yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: drone-server
namespace: drone
spec: 
replicas:  1
template: 
metadata: 
labels: 
app: drone-server
spec: 
containers: 
- image: drone/drone:0.8
imagePullPolicy:  Always
name:  drone-server
ports:
- containerPort: 8000
protocol:  TCP
- containerPort: 9000
protocol: TCP
volumeMounts:
# Persist our configs in an SQLite DB in here
- name: drone-server-sqlite-db
mountPath: /var/lib/drone
resources: 
requests:
cpu: 40m
memory: 32Mi
env:
- name: DRONE_HOST
valueFrom:
configMapKeyRef:
name: drone-config
key: server.host
- name: DRONE_OPEN
valueFrom:
configMapKeyRef:
name: drone-config
key: server.open
- name:DRONE_DATABASE_DRIVER
valueFrom:
configMapKeyRef:
name: drone-config
key: server.database.driver
- name: DRONE_DATABASE_DATASOURCE
valueFrom:
configMapKeyRef:
name: drone-config
key: server.database.datasource
- name: DRONE_SECRET
valueFrom:
secretKeyRef:
name: drone-secrets
key: server.secret
- name: DRONE_ADMIN
valueFrom:
configMapKeyRef:
name: drone-config
key: server.admin
- name: DRONE_GITHUB
valueFrom:
configMapKeyRef:
name: drone-config
key: server.remote.github
- name: DRONE_GITHUB_CLIENT
valueFrom:
configMapKeyRef:
name: drone-config
key: server.remote.github.client
- name: DRONE_GITHUB_SECRET
valueFrom:
configMapKeyRef:
key: server.remote.github.secret
- name: DRONE_DEBUG
valueFrom:
configMapKeyRef:
name: drone-config
key: server.debug

volumes:
- name: drone-server-sqlite-db
hostPath:
path: /var/lib/k8s/drone
- name: docker-socket
hostPath:
path: /var/run/docker.sock

Jenkins can be launched in much the same way. Because it is deployable in a Docker container, you can construct a similar deployment file and launch on Kubernetes. Here’s an example below. This file was taken from the GCE examples repo for the Jenkins CI server.


yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: jenkins
namespace: jenkins
spec:
replicas: 1
template:
metadata:
labels:
app: master
spec:
containers:
- name: master
image: jenkins/jenkins:2.67
ports:
- containerPort: 8080
- containerPort: 50000
readinessProbe:
httpGet:
path: /login
port: 8080
periodSeconds: 10
timeoutSeconds: 5
successThreshold: 2
failureThreshold: 5
env:
- name: JENKINS_OPTS
valueFrom:
secretKeyRef:
name: jenkins
key: options
- name: JAVA_OPTS
value: '-Xmx1400m'
volumeMounts:
- mountPath: /var/jenkins_home
name: jenkins-home
resources:
limits:
cpu: 500m
memory: 1500Mi
requests:
cpu: 500m
memory: 1500Mi
volumes:
- name: jenkins-home
gcePersistentDisk:
pdName: jenkins-home
fsType: ext4
partition: 1

Launching Jenkins is similarly easy. Because of the simplicity of Docker and Rancher, all you need to do is take the set of deployment files and paste them into the dashboard. My preferred way is using the Kubernetes dashboard for all management purposes. From here, I can upload the Jenkins files one by one to get the server up and running.

Managing the Drone server comes down to configurations passed when launching. Hooking up to Github involved adding OAuth2 tokens, as well as (in my case) a username and password to access a repository. changing this would involve either granting organization access through GIthub, or relaunching the server with new credentials. This could possibly hamper development, as it means that Drone cannot handle more than one source provider. As mentioned above, Jenkins allows for any number of source repos, with the caveat that each job only uses one.

Task 2 – Plugins

Plugins in Drone are very simple to configure and manage. In fact, there isn’t much you need to do to get one up and running. The ecosystem is considerably smaller than that for Jenkins, but there are still plugins for almost every major tool available. There are plugins for most major cloud providers, as well as integrations with popular source control repos. As mentioned before, containers in Drone are first class citizens. This means that each plugin and executed task is also a container.
Jenkins is the undisputed king of plugins. If you can think of the task, there is probably a plugin to accomplish it. There are at last glance, almost 1000 plugins available for use. The downside of this is that it can sometimes be difficult to determine, out of a selection of similar looking plugins, which one is the best choice for what you’re trying to accomplish

There are docker plugins for building pushing and images, AWS and K8s plugins for deploying to clusters, and various others. Because of the comparative youth of the Drone platform, there are a great deal fewer plugins available here than for Jenkins. That does not however, take away from their effectiveness and ease of use. A simple stanza in a drone.yml file will automatically download, configure, and run a selected plugin, with no other input needed. And remember, because of Drone’s relationship with containers, each plugin is maintained within an image. There are no extra dependencies to manage; if the plugin creator has done their job correctly, everything will be contained within that container.

When I built the drone.yml file for the simple node app, adding a Docker plugin was a breeze. There were only a few lines needed, and the image was built and pushed to a Dockerhub repo of my choosing. In the next section, you can see the section labeled docker. This stanza is all that’s needed to configure and run the plugin to build and push the Docker image.

Task 3

The last task is the bread and butter of any CI system. Drone and Jenkins are both designed to build apps. Originally, Jenkins was targeted towards java apps, but over the years the scope has expanded to include anything you could compile and execute as code. Jenkins even excels at new pipelines and cron-job like scheduled tasks. However, it is not container native, though it does fit very well into the container ecosystem.


yaml
pipeline:
  build:
    image: node:alpine
    commands:
      - npm install
      - npm run test
      - npm run build
  docker:
    image: plugins/docker
    dockerfile: Dockerfile
    repo: badamsbb/node-example
    tags: v1

For comparison, here’s a Jenkinsfile for the same app.


groovy
#!/usr/bin/env groovy
pipeline {
 agent {
  node {
   label 'docker'
  }
 }
 tools {
  nodejs 'node8.4.0'
 }
 stages {
  stage ('Checkout Code') {
   steps {
    checkout scm
   }
  }
  stage ('Verify Tools'){
   steps {
    parallel (
     node: {
       sh "npm -v"
     },
     docker: {
       sh "docker -v"
     }
    )
   }
  }
  stage ('Build app') {
   steps {
    sh "npm prune"
    sh "npm install"
   }
  }
  stage ('Test'){
   steps {
    sh "npm test"
   }
  }
  stage ('Build container') {
   steps {
    sh "docker build -t badamsbb/node-example:latest ."
    sh "docker tag badamsbb/node-example:latest badamsbb/node-example:v${env.BUILD_ID}"
   }
  }
  stage ('Verify') {
   steps {
    input "Everything good?"
   }
  }
  stage ('Clean') {
   steps {
    sh "npm prune"
    sh "rm -rf node_modules"
   }
  }
 }
}

While this example is verbose for the sake of explanation, you can see that accomplishing the same goal, a built Docker image, can be more involved than with Drone. In addition, what’s not pictured is the set up of the interactions between Jenkins and Docker. Because Jenkins is not Docker native, agent must be configured ahead of time to properly interact with the Docker daemon. This can be confusing to some, which is where Drone comes out ahead. It is already running on top of Docker; this same Docker is used to run its tasks.

Conclusion

Drone is a wonderful piece of CI software. It has quickly become a very popular choice for wanting to get up and running quickly, looking for a simple container-native CI solution. The simplicity of it is elegant, though as it is still in a pre-release status, there is much more to come. Adventurous engineers may be willing to give it a shot in production, and indeed many have. In my opinion, it is best suited to smaller teams looking to get up and running quickly. Its small footprint and simplicity of use lends itself readily to this kind of development.

However, Jenkins is the tried and true powerhouse of the CI community. It takes a lot to topple the king, especially one so entrenched in his position. Jenkins has been very successful at adapting to the market, with Blue Ocean and container-based pipelines making strong cases for its staying power. Jenkins can be used by teams of all sizes, but excels at scale. Larger organizations love Jenkins due to its history and numerous integrations. It also has distinct support options, either active community support for open source, or enterprise-level support through CloudBees But as with all tools, both Drone and Jenkins have their place within the CI ecosystem.

Bio

Brandon Adams
Certified Jenkins Engineer, and Docker enthusiast. I’ve been using Docker since the early days, and love hearing about new applications for the technology. Currently working for a Docker consulting partner in Bethesda, MD.


Continuous Delivery of Everything with Rancher, Drone, and Terraform

August 16, 2017

It’s 8:00 PM. I just deployed to production, but nothing’s working. Oh, wait. the production Kinesis stream doesn’t exist, because the CloudFormation template for production wasn’t updated. Okay, fix that. 9:00 PM. Redeploy. Still broken. Oh, wait. The production config file wasn’t updated to use the new database. Okay, fix that. Finally, it works, and it’s time to go home.
Read more


Winning Hackathons, DevOps-Style!

August 3, 2017

Recently, I moved to New York City. As a new resident, I decided to take part in the NYC DeveloperWeek hackathon, where our team won the NetApp challenge. In this post, I’ll walk through the product we put together, and share how we built a CI/CD pipeline for quick, iterative product development under tight constraints.

The Problem: Have you ever lived or worked in a building where it’s a pain to configure the buzzer to forward to multiple roommates or coworkers? Imagine that a friend arrives and buzzes your number, which is set to forward to your roommate who is visiting South Africa, and has no cell service. If you’re running late, your friend is just stuck outside.

The Product: We built a PBX-style application that integrations with Zang, and forwards a buzzer to multiple numbers, and even allows your friend to use a PIN on his phone to gain entry.

The constraint: For the hackathon, we had to use hardware that had already been setup and allocated.

 

Building our Hackathon CI/CD Pipeline

Our newly-updated eBook walks you through incorporating containers into your CI/CD pipeline. Download today!

For the competition, we knew that we wanted our builds scalable from the start, and that each deployment would snapshot our entire data environment pre-deployment using NetApp ONTAP (a sponsor of the Hackathon, and a really nice group of folks); if any deployment had an issue, we could simply and quickly roll back.

Fortunately, I am a firm believer in building stateless, Docker container-ready, rebuildable architecture; I can’t go back to the world that existed before: no real CI/CD, lots of SSH-ing into other systems, or SCP-ing data from one place to another. Thankfully, today we have tools and strategies that can help us.

Here’s what we used: Read more


Building a Super-Fast Docker CI/CD Pipeline with Rancher and DroneCI

July 5, 2017

At Higher Education, we’ve tested and used quite a few CI/CD tools for our Docker CI pipeline.  Using Rancher and Drone has proven to be the simplest, fastest, and most enjoyable experience we’ve found to date.   From the moment code is pushed/merged to a deployment branch, code is tested, built, and deployed to production in about half the time of cloud-hosted solutions – as little as three to five minutes (Some apps take longer due to a larger build/test process).

Drone builds are also extremely easy for our developers to configure and maintain, and getting it setup on Rancher is just like everything else on Rancher – very simple.

 

Our Top Requirements for a CI/CD Pipeline

The CI/CD pipeline is really the core of the DevOps experience and is certainly the most impactful for our developers.  From a developer perspective, the two things that matter most for a CI/CD pipeline are speed and simplicity.

Speed is #1 on the list, because nothing’s worse than pushing out one line of code and waiting for 20 minutes for it to get to production.  Or even worse…when there is a production issue, a developer pushes out a hot fix only to have company dollars continue to grow wings and fly away as your deployment pipeline churns.

Simplicity is #2, because in an ideal world, developers can build and maintain their own application deployment configurations.  This makes life easier for everyone in the long run.  You certainly don’t want developers knocking on your (Slack) door every time their build fails for some reason.

 

Docker CI/CD Pipeline Speed Pain points

While immutable containers are far superior to maintaining stateful servers, they do have a few drawbacks – the biggest one being deployment speed:  It’s slower to build and deploy a container image than to simply push code to an existing server.  Here are all the places that a Docker deployment pipeline can spend time: Read more


How to Run GitLab in Rancher - Part 2

June 22, 2017

GitLab and RancherThis is part two of our series on using GitLab and Rancher together to build a CI/CD pipeline, and follows part one from last week, which covered deploying, configuring, and securing GitLab in Rancher. We’ve also made the entire walkthrough available for download. 

Using GitLab CI Multi-Runner to Build Containers

GitLab CI is a powerful tool for continuous integration and continuous delivery. To use it with Rancher, we’ll deploy a runner that will execute jobs.

Launching the Runner

There are several ways that runners can be deployed, but since we’ll be targeting building containers from our repositories, we’ll run a Docker container that has direct access to /var/run/docker.sock to build images that are siblings to itself.

  1. In Rancher, add a service to your Gitlab stack
  2. Set it up with the following configuration:
  • Name: runner01
  • Image: gitlab/gitlab-runner
  • Console: None
  • Volumes:
    • /var/run/docker.sock:/var/run/docker.sock
    • runner01-etc:/etc/gitlab-runner

When the container launches, it will create a default configuration in /etc/gitlab-runner, to which we’ve connected a volume. The next step is to register the runner with your Gitlab instance.

The options that I’m setting below are correct for a basic runner that will build any job. You can also limit runners to specific repositories or use other images. Read the documentation from Gitlab to learn what options are best for your environment.

Read more


How to Run GitLab in Rancher - Part 1

June 15, 2017

Note: This post is the first in a two-part series on using GitLab and Rancher together for continuous integration and deployment, and part two is now up. We’ve also made the entire walkthrough available for download

GitLab and RancherIntroduction

GitLab is, at its core, a tool for centrally managing Git repositories. As one might expect form a platform that provides this service, GitLab provides a robust authentication and authorization mechanism, groups, issue tracking, wiki, and snippets, along with public, internal, and private repositories.

GitLab goes further by including a powerful continuous integration (CI) engine and a Docker container registry, allowing you to go from development to release inside of the same utility. It continues by adding two more powerful open source software utilities:  Prometheus for monitoring, and Mattermost for team communication. The platform has a solid API and integrates with existing third-party systems like JIRA, Bugzilla, Pivotal, Slack, HipChat, Bamboo, and more.

All of this begs the question: why use GitLab instead of using SaaS services directly? The answer lies in personal taste. For most people, paying SaaS providers for the services that they provide is an excellent solution. You can focus on building your application and let them worry about maintaining the tools. What if you already have infrastructure with spare capacity? What if you just want private repositories without paying for that privilege? What if you did the math and determined that you can save money by hosting it yourself? What if you just want to own your own data?

Bringing services in-house opens you up to the risk of spending more time managing them and getting them to talk to each other, which is why GitLab truly shines as an option. With one click and a few minutes of configuration, you can be up and running with a complete solution.

Read more


Upcoming Events


Recent Posts