We finally have some basic CI/CD pipeline for our containers running on top of a Mesos/Marathon stack ! I'm taking this opportunity to quickly share what's been done this week.
We are using Jenkins to build a generic Play! application (which implements both my API tier as well as the long-lived actor system performing Kafka streaming). Now the hiccup is that we need those artifacts to land into a Docker image but also to be re-deployed gracefully in several clusters (one per region typically). The second part is a bit less trivial since we are talking about orchestrating a deployment over many live containers and making sure everything is gracefully phased out, etc.
It is clear I just wasn't ready to invest into developing some over-complicated Jenkins jobs and end up in a maze of build steps. Jenkins is very effective as a tool but you really want to keep things simple.
I did split the problem in half: the artifacts are built on any regular Jenkins slave living happily within our internal network (and having access to our git). The rest has to be executed on specialized slaves that run within the Mesos cluster. Why ? Simply because I can then run them as Ochopod containers meaning they will automagically hook up with the proxy and from then on be able to issue elaborate commands (switch such and such pod on, deploy a new image, phase out this or that, etc.).
The nice thing with that approach is that a) it is extremely self-contained, b) it is all managed via the Ocho-* CLI and c) I can offload all the programmatic logic into my slave (and code in Python). All the credentials, keys and what's not you need to deploy your stuff can also be centralized in there.
I came up with a basic containerized slave which is packaged for our Ochothon mini-PaaS and runs sshd. You can peruse the repository but the things to remember are: you can issue arbitrary commands to your local proxy (the slave will CURL them for you) plus you can docker build/push images (we mount the .dockercfg from the slave and socat the daemon's unix socket to perform REST calls). Those capabilities are offered via a tiny Python package installed on the slave and which can be augmented easily with higher-level tools.
An artistic view of the whole system looks like:
So here we go. I simply spawned a new slave container using my CLI:
What's TCP 22 mapped to and where is the container running from ? The CLI knows it all !
OK, simply setup a new Jenkins slave with that IP and port (the firewalling around your Mesos cluster must of course allow Jenkins in, duh).
Poof, launch the slave, SSH into the container and see if it is running (notice the socat that the container runs as a way to communicate to the underlying Docker daemon):
A little remark: the label system in Jenkins is quite cool and allows me to cherry-pick specific slave containers depending on what region/environment I need to work on. For instance I just run a single slave for all my us-east pre-production environments.
Great I can now happily get Jenkins to run crazy commands and orchestrate stuff !
Before we do so let me explain how I did setup my build numbers and what ultimately is used to tag my Docker images. I use 2 build jobs: the actual compilation dealing with my Play! application and then the Docker build/push that happens in cluster. The first job triggers the second and we copy its artifacts over. I also write out the Play! build number into a little TAG file in the workspace. This is then used by the second job to tag the Docker image (we don't really care about about sequencing from anything other than the actual build job). This simple trick is way better than trying to tinker with environment variables, plugins or even CURLing Jenkins...
My basic Jenkins slave image contains a spiffy Python script called dockerize which will perform the build/push by using the underlying daemon's REST API. It looks like this in practice:
So just to recap this script is just some Python code bundled with my container and leveraging the Docker daemon running underneath. I just simply specify it as a vanilla shell script in my Jenkins job configuration. What about the target registry credentials you will ask ? Well we mount the .dockerfile found on the slave and use it to craft the proper REST call for the push. Cheap but effective.
Once I get my image published to our Docker Hub account what do I do ? I very simply use the same mechanism to trigger a rolling-push from my containerized Jenkins slave ! As you may remember our Ochothon PaaS proxy is equipped with lots of useful things including a deploy tool that will spin new containers up and then phase the old ones out (and yes everything is done gracefully with no service interruption).
Let's use that from Jenkins ! I did write a little script that will talk to the proxy, use deploy with a configuration that references the latest stable build and check for results (keep in mind I have to deploy two specific tiers to add to the fun).
The corresponding Jenkins job for, say, my development environment running on the west coast is deliciously compact. The following snippet will effectively perform a rolling re-deployment of my two tiers one after the other and use the specified tag to pull from the Hub:
You can go as crazy as you want and also incorporate high-level acceptance tests, etc.
Bottom line: if you can spare room to run one slave container per cluster you then have a terrific option to trigger programmatic deployment logic straight from Jenkins and thus avoid ending up with unmanageable jobs featuring twisted combinations of interlocking steps.
Have a look at our little sample slave and don't hesitate to reach out as usual !