“What is a name? That which we call a rose by any other name would smell as sweet.” - William Shakespeare
What is with the name Ochopod you may be wondering? As with all successful businesses and bands, it all starts with a name. For example, would the band Hooba Stank been as successful if they were called the Harmonizers? Well, with all good names comes the story behind it.
It all started when I had to deliver our event streaming infrastructure which was in itself complex with a bunch of moving pieces comprised of: Zookeeper, Kafka, HAProxy, RabbitMQ, two web-serving tiers, etc... The time to deploy and maintain all those systems alone would have been impossible for me to maintain and the overhead to get others ramped up to manage all those systems was a non-option. Then to add to the complexity, let’s have this system run across multiple regions with replication. This is a pretty daunting task, right? Hence, the engineer in me knew that there had to be a more efficient and practical solution for accomplishing this.
So with much head scratching, back of the napkin drawings, it hit me… I had been already using Docker quite heavily, loved it but was wondering how I could bridge the gap between configuration, deployment, orchestration and self-healing? Then it came to me, I decided on a Apache Mesos, Docker, Marathon stack. Why might you ask? Well, the setup is dead simple, nicely resilient and most importantly, does not get in my way. The API is very light and generic. I just have to set it up using Chef and from then on I am working only with containers.
Mesos and Docker... Okay but then what? Now, the last thing I wanted to do was to rely on yet another external controller to orchestrate my containers. I wanted instead to just drop them into my cluster and “voila!", have the whole thing all configured automagically. This of course meant that all the cluster aware configuration bits had to be located in-container.
Based on these new considerations, I came up with good idea #2, “have one unique layer dealing with cross-container configuration.” This layer would be embedded in each container, be reasonably light and not tied to how the underlying stack is composed. Python was the obvious choice for implementing this, adhering to my favorite mantra of easy to setup, super flexible and full of amazing packages such as Requests, Pykka or Kazoo.
Wait, how would you then cluster stuff up? Why not have all our containers elect a leader and once everybody is up and running have the leader coordinate a configuration sequence. Such a protocol is already enough to let you shape precisely how your containers should be configured at runtime, which differs a lot depending on what you intend to run as you probably know. How would the leader know about the other containers? By having everybody registering to some shared key-value data store, of course! So in the Mesos case I simply piggy-back on the Zookeeper ensemble used by the masters. Easy-peasy.
So what do we have? We now have a single Docker image that runs the same code. Each container we instantiate then registers who it is, where it runs from in Zookeeper and attempt to grab the same lock. The container that gets that lock is the leader. All our containers also listen on a small REST/HTTP Flask endpoint for requests. The leader’s job is to constantly monitor for changes (a new container registering or maybe a container going down). Any change causes the leader to undergo a configuration pass: a request is sent to each container in order (including the leader itself).
The configuration logic is the same, of course, on each container but can leverage a holistic view of what the cluster is (e.g. you know about all the other IP addresses and remapped ports). You finally specify what it is you want to run and that’s it. There you go, you start with one image, start a Marathon application, scale it up and you end up with a cluster of containers that can coordinate without you in the picture. You can then leverage the semantics offered by your framework and happily set quotas, placement constraints and so on.
Configuration? The Dockerfile defines what I need to download and install.
Clustering? My boot script defines what process i want to run and how it should be dynamically setup.
Deployment? Mesos and Marathon do the job perfectly well.
A 10 pieces distributed system that can scale and be deployed in just one single click? Not an issue anymore!
So have you figured out why we called this solution, Ochopod? If not completely obvious now, the term Ochopod is in reference to a super intelligent cephalopod, highly adaptive, eight tentancled creature. A perfect symbolic representation of our Container strategy with built in intelligence, dynamic orchestration, and ease in adaptability.
Don't forget to check our cool documentation for more details!