We recently had the pleasure to exchange ideas with the great folks at Kismatic during our last meetup at the Autodesk Gallery in SF. I was asked if our Ochopod technology would run on Kubernetes. My answer was subtle at usual : "uh well yea uh sure why not dee dum". It was then clear I had to wake up and do something about it.
It took around 8 hours to get acquainted with Kubernetes (knowing very little to start with). Patrick (the CEO @ Kismatic) had told me it installed super fast but boy was it easy. A simple one liner and poof I had my little cluster to play with on EC2. Clearly a win for Kubernetes already.
So, was it really that different from a Mesos based stack ? Yes and no. The controlling semantics were roughly the same but what stood out for me was the very cool IP mapping Kubernetes does. One set of co-located containers maps to its own IP without explicit need for remapping ports. Definitively quite elegant.
In the Ochopod world I deal with sets of containers forming virtual clusters. These containers are managed by the underlying stack using whatever semantics it offers. For Marathon I simply use their notion of application. In the Kubernetes case I naturally picked the replication controller: one Ochopod cluster then maps to one or more controllers. The Kubernet-esque notions of DNS and services are also extremely appealing and could open more possibilities. I honestly did not even have time to dive more into the vast amount of features the system offers.
Now, what about our leader election ? Remember Ochopod synchronizes its clusters using Zookeeper. In my initial Mesos implementation I simply piggy-back onto the ensemble used by the masters. With Kubernetes I either re-implement the core to deal with, say, Etcd or I run some Zookeper myself. I felt lazy and picked the second solution.
Ok, good. So I can come up with Kubernetes bindings, cool. Now what ? Am I going to write yet another CLI layer to deal with a new API ? Mmh, that didn't feel right (let alone that you won't reach into the container IP space from the outside world). After all Ochopod is supposed to make your life easier. Well then let's use some man-in-the-middle, a proxy of some sort. And hey why not co-locate it with a Zookeeper instance ?
You get the picture and see what I came up with: a simple Kubernetes pod that acts as a Ochopod proxy and brings all you need to get started. I already had a bunch of code done for that meetup demo (including a simple but effective web-shell courtesy of JQuery). You can go look at the whole thing in github.
So, do I like Kubernetes ? Did my friends at Kismatic lie to me ? No they did not and Kubernetes is indeed an truly amazing piece of technology.
Now am I going to scrap my Mesos stuff at this point ? Probably not either. There is clearly space for both systems to co-exist depending on your dev/ops capabilities & goals. But having said that things like having a consistent addressing space or an internal DNS are clearly a huge win I would love to see one day in Mesos.