After a long day of dev/ops to inaugurate the latest Ochothon release I though I'd share quick what it feels like deploying our internal event streaming system over Mesos/Marathon. This piece of infrastructure is running in production as I write this post.
I sticked to my early Marathon/EC2 deployment method (Chef based) but could totally switch over to DCOS from Mesosphere already. K8S over GCE is also a tempting option for the medium term.
Anyway, what do we have ? Well we have nice mix of things : a central set of Kafka brokers and its companion Zookeeper ensemble, a first API tier (Play! framework, 100% Scala), a central RabbitMQ "phone-switch", a second tier handling long-live streaming actors (Play! + Scala again) and a little Redis instance holding the whitelisting & credentials. The whole thing is front-ended by HAProxy.
This blob (which I'll describe in details soon in another post) is deployed in each region and we slap on top additional containers performing automated region-to-region replication. Phew !
Each piece in the puzzle has its own Docker image. Well its own Ochopod image should I say as we package both the software we need to run (Redis for instance) and its runtime logic (how to configure it in a cluster, monitor it and so on).
Please note we do not run any framework other than Marathon ! That is the full system is containerized using Ochopod and that's it. Much simpler if you ask me.
At this point all I need is a Mesos/Marathon cluster onto which I deploy my Ochothon proxy. That proxy exposes a handy remote shell equipped with all the tools you could ever dream of. Let's check the proxy is ready using my little CLI :
The #3 means this is the 3rd "portal" container I deployed on that Mesos cluster. That portal is what the CLI talks to by the way as there is no tooling installed on my local machine.
Before I continue let me illustrate how we deploy stuff with Ochothon. We use a high level YAML definition file that my CLI understands and which is a generic (read "simplified") version of what the Marathon (or K8s) API expects.
Stuff like port mappings, arbitrary parameters and so on are exposed neatly (and turned into what the actual API expects by the deploy tool). You can also add verbatim data, for instance for idiosyncrasies such as placement constraints or quotas. As an example the definition for my Redis container looks like :
Allright, here we go: let's test and deploy a 3 node Zookeeper ensemble and 4 Kafka brokers to go with it !
The Zookeeper containers will form a cross-configured ensemble while the Kafka containers will attach & format their own volumes automatically. Let's see if everybody is up and running:
Wondering what that -n test option is ? You can namespace your Ochopod containers, which is a great option when running multiple systems off the same Mesos cluster. Right now my stuff lives in the test namespace.
You will also notice some containers will give you overall status information which is extremely handy when dealing with Kafka for instance (I run high-level health check periodically and will detect stale brokers and leader-less topics).
OK, same drill. Let's slap Redis, RabbitMQ and HAproxy:
Oh wait, why is HAProxy showing up as "configuring" ? Well this is because it is dependent on our API tier which is not there yet. Yes, Ochopod manages transitive dependencies for you !
Last but not least, let me add my 2 Play!/Scala tiers. Let's do just one of each to test quick:
OK, now in theory our HAProxy will start configuring itself and point to our API container. Let's test it by first looking where it is running from:
Let's curl it using its health GET /ping endpoint:
Boom it works (the "n/a" is supposed to be the build number which I haven't set yet). That's it. Mission accomplished !
Let's have a final glance at the underlying Marathon applications that were deployed for me by the remote Ochothon toolset (each application maps one of my clusters) :
This is pretty much it. I just deployed a real-life distributed system involving 6 technologies in approximately 2 minutes. Now you are allowed to ask the questions that tormented you all along: why not use the existing Kafka framework ? What's better with your stuff ?
Well I will just point out one thing: if you now scale up or down any container in my backend the whole thing will automatically adapt and re-configure itself. Period. Same goes by the way when rolling out new builds for my API tier: simply push a new set of containers, wait for configuration, phase out the old ones (blog post coming up soon about that too).
Feel free to reach out to us if you feel that Ochopod can help you build better distributed backends that deploy faster !
PS : OK i'm now done with my stuff. Let's tear it down in one go. Poof, gone !