Quick note to explain how I shape most of my code based on automata.
I often face the same old situation where I deal with iterative logic that can fail. I know this sounds stupid but take for instance some code that needs to perform in sequence some API calls against a remote REST API. Take another simple example and picture a process that receives events and must send them in batches to some processing backend. What I truly hate is having to deal with potential failures without messing up the code and ending up with some maze-like code.
You usually fall into either mess-ification (e.g code that nobody else will be able to maintain) or over-engineering by building some complex framework that will end up being equally messy. Alternatively you can attempt to go full functional with non-zero probability none of your co-workers will be ever able to decipher what you meant to write.
In order to boost coherence and (somewhat) improve clarity I experimented with finite state-machines. Without getting into the details we are talking about a simple automaton that flows between a set of states. This is widely used in many engineering applications and can be very effectively used in our case! There is a lot of theory as well but what I'm trying to get to is actually really trivial.
What I want is to be able to define a state-machine that has an initial state, a reset state and one or more processing states. Each state simply leads to the next one and any failure (explicit asserts or unexpected errors) trips the machine back to the reset state. This works like a one-fits-all sink that will guarantee you use one single code-path to perform stuff like cleaning-up variables or resetting a network connection.
My earlier example describing a process batching events can then very nicely be turned into an automaton that maintains an input FIFO and defines a single spin state. The thing just dequeues one event at a time from the FIFO, processes it and loops back to that spin state. If anything bad happens (typically some downstream failure) I can simply assert & trip the automaton to reset it before looping back to the initial state. You can't get really simpler than that.
Actor based frameworks are perfect to implement such automata in a concise way. Look at the awesome Pykka framework for Python or of course Akka for those doing Scala and/or Java. An actor can be turned into an automaton without much work thanks to message passing. Each state can be implemented as its own message for instance. Once we're done with a state we just fire the next one to ourselves. You can also pass contextual information between states that way. Another nice thing is that you can also couple more several automata and have them exchange messages.
The ROI is actually substantial in my experience: less code combined with stronger waterproofing, ability to capitalize on the actor semantics, increased parallelism and so on. Of course you need to re-think your logic a bit to map it to one or more automata. The boiler-plate code required is really minimal. As an added bonus you can also inject some good patterns in the mix .. for instance the circuit breaker. Things like automatic re-tries are also handled perfectly with minimal code.
Enough talking. For those with a curious mind I have a little Pykka state-machine for Python. I use this a lot in Ochopod where the core logic is implemented as an interlocked automata pair talking to a Flask web-server. Feel free to look at it.
You can also find an actual example involving the state-machine: the automaton I use to perform a periodic diff in the metadata I store in Zookeeper. In that case a disconnect will make the code throw at which point the automaton will gracefully go down.
Happy coding !