Mars Rovers in Clojure - Rewriting an Akka project to a Core Async one

I've decided to to reimplement Kaloz's Mars Rover Scala (for brevity's sake SMR from now on) project in Clojure (CMR). SMR is based on Actors of the Akka framework. Actors are the manifestation of the original idea of object orientation, that is, independent entities hiding their internal state and communicating via message passing. Since the whole model is purely about side effects, it seemed a nice challenge to implement in a functional way.
I wanted to have a jar as an end product that can be substituted to the backend of the original Scala, using the original presentation layer. I'm not there yet, but a first version is available on Github, where mars rovers can run in the REPL, spitting out their positions simply to the console.
All in all the project is a nice round 600 LOC. The Scala version had 548, which is not surprising. Scala, much like Clojure, is a very concise language, but because of its syntax you can compress the code more and still retain readability.

The main differences between SMR and CMR are

• Instead of complecting state, identity and behavioural logic, the Clojure version uses pure functions to capture the behaviour.
• All the "actors" have a receive method that takes the state of the "actor" and the received message (plus some additional information sometimes) and returns the new state with possibly a list of messages and executable effects. 
• Context changes in actors are implemented in a surprisingly elegant way via multimethods. Different version of receive is called based on the state of the "actor"
• The actors are simple atoms. The state of the atom is the input of the receive and the state part of the output is swap!-ed into it
• The Clojure version, for the time being, lacks the distributed nature of Akka, and also its error handling mechanism. 
• It uses the core.async for asynchronous communication. Where in the Scala version actors hold references to other actors, here channels are passed around
• The architecture, not surprisingly, is quite different

Application = 75% pure functions + 6% glue + 8% application-level + 11% other

During the reimplementation I dissected the code to form 4 distinct parts. LOC-wise the sizes of these parts are

1. 75% is pure, framework-neutral functional code. I moved all those source files under marsrover.pure package. 
2. 6% is the glue, a mini messaging-framework I built over core.async in marsrover.glue.clj. In this "framework" a component's state is preserved in an atom. The component has an in-channel, an atom, and a function defining its behaviour. When a message arrives on the channel the component's state (value of the atom) and the message are the inputs of the behaviour function, and the output is the new state, the messages it has to send out, and possibly some effects. It's all very high level, doesn't know anything about our particular domain. 
3. 8% is the application layer, it's in marsrover.app.clj. This takes the pure part and injects it in the glue. 
4. 11% is other stuff like reading the expedition config (number of rovers, size of the plateau, etc).

So there is a big bulk of pure functional blob and a very thin layer glueing it together to form a proper application. I find this really fascinating. In the original Scala/Akka implementation the Akka framework and the logic was undisentangibly intertwined. Now the glue uses core.async, but if I wanted to use an actor-framework instead or any other approach, I would only need to change that 35 lines confined in a single source file. I say it again, it's fascinating. With functional approach we can push the framework/technology -related code to the outmost layer of our application, making it seem almost insignificant. The pure functions won't change if we want to move from the asynchronous model to synchronous, or replace core.async with any other integration solution. Imagine you'd wanna rewrite Mars Rovers in Scala, but not using Actors.

Finally, you can run the app by running marsrover.mars_expedition.clj in the REPL.

New module added to Principle

I've added a new module to JPrinciple recently to enable the developer to constrain access to third party libraries to designated parts of the code. The reason for this is that I sometimes see processing json in app layer or using the Mongo API from the domain in our codebase. Being an ardent DDD proponent, I just cannot have that happening again! So the idea is that you can specify which libraries you allow access to in which layer. All the layers around the specified one can use those libraries of course, but nothing under it. Hope it made some sense. Probably it's better to see through an example


The new part is the <thirdparty>...</thirdparty>. It says that anything under org.apache.commons can be referenced in the Domain layer and up (app and infrastrucure), and the Infrastructure layer can use anything under the listed package paths: org.apache.maven,org.json,org.yaml,com.google.common.collect,jdepend. Anything else used in Infrastructure, e.g. something under org.mongo, would cause an alert. So would the reference to org.json in Domain, since it's only allowed in Infrastructure.
The visual idea behind it is that your code is a castle with multiple circles of defending walls around it. You let foreign troops leaking into your territory only until certain walls. Different troops can have different privileges, one (org.yaml) can only set up his tent inside the outmost wall (Infrastructure), the other (org.apache.commons) is allowed to enter the inner sanctum (Domain). Usable from version 0.25.

Clojure, syntax and readability

def isValidVoter(p:Person) = p.age > 18 && !p.isInPrison

vs

(defn is-valid-voter [p] (and (> (:age p) 18) (not (:in-prison p))))

The first expression is in Scala, the second in Clojure. As much as I appreciate the simpleness of Clojure-syntax, I can't help but find the Scala version much easier to read. The Clojure version can be enhanced by line breaks

(defn is-valid-voter [p]

  (and 

    (> (:age p) 18) 

    (not (:in-prison p))))

but now it's a four-liner instead of a one-liner. And yet still less readable than its Scala counterpart. Furthermore, if I want to remove the "in-prison" part, in Scala I just remove the end of the expression from the && sign without touching anything from the left of it.

def isValidVoter(p:Person) = p.age > 18 && !p.isInPrison

In Clojure I have to remove the 'and' clause, thus changing the number of parantheseses before and after  (> (:age p) 18). This is not directly related to readability, but also the result of Lisp-syntax.

Back to the question of readability, part of it is due to our learnt preference for infix notation. But the real reason is, I think, is that the different parts of the function definition, such as name, arguments, body, just don't stand out in Lisp. Check this

(defn wrap [x] [x])

This little function does nothing but wrap the input in a vector. The argument list and the body are 100% identical. Even after some exposure to Clojure it can look puzzling for a second. In Scala the two sides of the '=' sign are obviously stand apart. Syntax highlighting, but the syntax itself makes it easy capture the whole in one glance, exactly because the syntax is rigid and there are rules it has to abide. Lisp on the other hand has a syntax so simple we can almost say has no syntax at all. Everything is a list (S-expression to be exact), the first element is the function name, the rest are its arguments (let's forget about special forms and macros for now). That's it.
It lacks arbitrary rules like "there is a '=' between the expression signature and the body", "prefix a boolean expression with an '!' negates it" or "get the field of an object with <object>.<field>", and so on. Taking uniformity to its extreme, undeniably some readability is lost.  But what is gained is not only the esthetic statisfaction over the ultimate simplicity, but the power coming from homoiconity, macros. Macros are discussed elsewhere, but I'm as much as fascinated by the simplicity itself as by the power it brings. There are many syntaxes with different advantages and disadvantages. Languages are evolving and we can't know how a programming language will look like in a hundred years. But Lisp is on a very end of the spectrum. It cannot get any simpler.

OO, invariants and Clojure

In an object-oriented language if we'd like to implement a "concept", we create a class. Concept is too abstract (and probably there are better words out there for what I tried to say), so let's see a practical example. I want to implement a game of cards and therefore I'd like to deal with "decks". My OO-language of choice is Scala. Assuming that the operations I want to do on a deck are pulling a card from top, shuffle, and adding cards on the top, the code would look like this: (ignore Scala's awkward way of simulating enumerations - why is that so bad in this otherwise beautifully concise language is beyond me).
I chose to implement the deck as an immutable objects, all functions on it creating a new instance and leaving the original intact. I could have done the more OO-way of mutable state, but it's irrelevant for the case and I'd like to move the solution close to the one of Clojure discussed later.
See the check in the "constructor" and the isValidDeck function. This ensures that our Deck is always "valid". All the code using this class can safely assume that it doesn't contain duplicate cards. It's the simplest invariant I could come up, but serves the purpose of the post well. If I try to get out of my default OO-mindset acquired in the last 10 years and look at it with fresh eyes, what I see that we have a "raw" data structure, a list, and certain constraints attached to it. The data structure and the constraints together are the class.

Class = raw data structure + constraints

The state-space of the Deck object obviously couldn't be bigger than that of the list, the raw data. So building a class around a raw data structure is simply imposing restrictions on how the underlying data can be modified by the clients of the class. Of course the methods of a class can hide side-effects, too, but again, in the aspect of invariants, it's not relevant.
The benefit of the OO approach is that the Deck stands guard over its invariants, so the constrains and the data "travel together". The disadvantage is that we lose all the already existing data-manipulating functions the language would provide out of the box. Let's demonstrate it with the Clojure solution.

Delightfully simple. We didn't have to write any of the required functions, the language gives us them all and plenty more. What if we want to remove all the Clubs from the Deck? Or halve it? Or order by some specific rule? Clojure would offer a million different data manipulation choices. So would Scala anyway if you choose to use it that way, but the language philosophy prefers encapsulating data and behaviour in one unit (class).
The disadvantage of Clojure's lightweight way is that we now lost the guard on the invariants. You can pass an arbitrary list to any functions expecting a valid deck, opening the door for hard to detect bugs. However I think there is a remedy for this and quite an easy one of that.

Using the 'with-invariant' function we can apply any validation with any function operating on any data structure. We can extend it easily to deal with multiple inputs, too, or different validations for the input and the output. Object-oriented programming ties validation and data together, offering very strong safeguards, but constraining reusability immensely.

Classes vs (raw data + validator functions) = safety vs reusability

Imagine now we want to extend our game of cards program by allowing different kind of games, including ones where the deck can contain duplicates. Or requires different operations on the deck (like pull from the bottom). Can we reuse the Deck class? Not in this current form. Maybe subclassing it or extract some traits? Either way the code starts to get complicated.
With Clojure we just use the same basic data structure (list) a new validator function with with-invariant or build it up in one single function for brevity's sake. The Clojure implementation dissected the class and cleaved off the constraints from the data. The data structure can processed in almost every imaginable way with the rich set of functions built-in the language, and the constraints can be plugged-in the computation whenever needed.

Implementing a DDD project in Clojure - 2

One important thing I forgot to mention (even realize) in the first post is the applicability of the Dependency Inversion Principle for Clojure. In a DDD application adhering to Hexagonal Architecture the Domain layer is at the bottom of the dependency-chain. It has interfaces (Secondary Ports) implemented (as Secondary Adapters) in the Infrastructure layer, so the direction of dependency is the inverse of the direction of control-flow. In Clojure you can achieve the same with Protocols, but without application context and objects we need a Namespace (I called it registry) to instantiate the implementations and the Namespaces that use the Protocol have to depend on it. So through the registry the high-level Namespaces depend on the low-level ones. I don't know yet how to resolve it, if it's doable at all.

Clojure - " java.lang.Exception: Cyclic load dependency"

I've just discovered that Clojure, by default, doesn't allow cyclic dependencies between namespaces, as opposed to Java, where those between classes are allowed. Cyclic dependencies being the root of many evils, that is a very good thing. It would be nice to find the same strictness imposed on package levels, but it's not.

Implementing a DDD project in Clojure - 1

In the last couple of weeks I've been occupying myself with porting the Blackjack Java project to Clojure. I've been curious whether DDD concepts could be reused in a functional language without sacrificing idiomaticity. There is still a long way to go, but the walking (limping) skeleton is ready. Check it on github. In the followings I make an attempt to summarize my experiences.

1. Aggregates

First of all Clojure strongly encourages working with immutable data structures. My naive assumption had been that an OO aggregate can be simply substituted with a data structure accompanied by a handful of functions operating on it. This is only partially true. Though I've found state mutation easily replaceable with creating a new data structure (Clojure's persistent data structures make it performant, and its very effective data manipulation functions easy) as the output of the applied function, but I'm not convinced about how invariants should be preserved in the absence of an entity with mutable state. To some degree it can be achieved by :pre and :post constraints on the individual functions, but I haven't really dived into it.

2. Domain events

Related to the previous point, since in OO aggregate roots are to publish domain events. Initially I implemented my "aggregates" the same way, the functions operating on them published events as a side-effect. But eventually I've realized it's very non-FP, so instead of invoking a side-effect, I changed these functions to return not only the updated data structure (the new state of the "aggregate"), but the event-list as well. Then the new "state" is updated in the DB and the events are sent out. This solution (which could be applied in OO as well) eliminates the need of thread-local/event store-flushing magic I resorted to in the java version. I am more than satisfied with this, it's not only idiomatic, but makes things so much simpler.

3. AOP for locking

This is quite pleasing also. Instead of aspects and annotations, it's a 5-(short)-line macro.

4. Layering

The most interesting part. In the very few books mentioning the large-scale structure of FP-programs I've stumbled across the notion of onion-layering, not unlike that of DDD or Hexagonal Architecture. We should strive to confine the mutable states to the outer layers, keeping the core completely functional. Unfortunately in OO-DDD, the "core" is the Domain, which does have logic with side-effects. Repository interfaces and Domain Services as front-doors to ACLs are very much part of the Domain, and are there to enable the Domain to interact with the external world (writing to DB, sending messages, calling web services, ...).
So quite early it became clear that the layering in FP-programs requires some changes in the OO-way of thinking.

4.1 Hexagonal Architecture

If we forget about DDD temporarily and reach back to HA (which I think is one of the foundations of the former), some parallels with the FP-version of onion-layering are obvious. In HA, the Adapters are outside of the core and the Ports are the joints that bind the Adapters to the core. Even an FP-program must interact with the external world (otherwise what's the point of writing it), so HA, with its explicitly designated areas of external interaction seems to be a natural fit.

4.2 Application layer

Having a clean facade for an application has nothing to do with OO or FP, so keeping this layer of DDD seemed ok. In addition to the normal DDD application layer responsibilities, such as locks, transactions and orchestration, I decided all calls to secondary (driving) ports should happen here (calls to a remote Bounded Context, Blackjack Wallet, interacting with the DB). This is the external layer of the onion (still beneath the adapters-layer, though).
I've kept different namespaces in this layer for the different modules (Player, Table, Game) and used most method names for commands with bangs.

4.3 Core (Domain) layer

A purely functional layer. I've tried to simply port the java version's Domain here, minus the secondary (driving ports), e.g. the Game-, Player-, TableRepository or the WalletService, which I've decided to put under a port package on the same level as app and domain. Secondary ports neither defined (but in their own package) or used (but in app services, see previous point) here.

4.4. Infrastructure layer

Quite the same as in the java version. Adapters are implemented as Records of Protocols, see next point.

5. Using Protocols for secondary ports

I've tried to avoid OO-techniques as much as possible, but protocols simply seem to be the right way to implement ports.

6. Application Context and Dependency Injection

Although in the absence of objects FP doesn't require DI the same way as an OO program would, I've found the use of Protocols has similar needs to that of objects. The Protocol implementation has to be accessible to the function that uses it without explicit dependency on it. In a Java-Spring application the application context would do this wiring, as well as instantiating the proper implementation based on the content of a property file. Lacking any framework I created a dedicated namespace for it, called registry. All namespaces where functions need to call Protocols import the registry namespace. I don't know how others do this, but this solution seemed straightforward.

7. Modules

I kept the module structure of the Java version intact. Game, Table, Player and Cashier are the same, they interact with each other in an asynchronous way. Clojure Async does this job perfectly. There is a channel for Domain Events and subscriptions for the event handlers. Much shorter and cleaner than the Java version's was.

To be continued...