I've read a lot of interesting stuff about swarm-intelligence and a couple of weeks ago the idea suddenly came to implement something in Clojure. The full source code is in Github (the application can be run from the app.clj), here I'd like to show how the story has evolved step by step. The first part of the task was to implement some kind of "vector algebra" to model the moves of entities.
Thursday 27 November 2014
Monday 24 November 2014
Procuder-consumer example in Java and Clojure
I want to implement a very simply consumer-producer app. One consumer consuming from a large number of producers. I would like to see how much throughput is possible in accordance with the number of producers.
The first implementation is in Java
The Clojure version
The Java one is really struggling under 100 producers. I haven't done any tuning on the ExecutorService, though, but in the Clojure version I didn't even need to. The latter is distributing the load extremely evenly and reaches a much higher throughput than the Java version.
The first implementation is in Java
The Clojure version
The Java one is really struggling under 100 producers. I haven't done any tuning on the ExecutorService, though, but in the Clojure version I didn't even need to. The latter is distributing the load extremely evenly and reaches a much higher throughput than the Java version.
Thursday 6 November 2014
Dynamic typing - Timeout
Another piece of useful functionality made easy by dynamic typing and clojure.async. Adding a timeout to a function.
Tuesday 28 October 2014
Mars Rovers in Clojure - iteration 2
In the last couple of weeks I spent a fair amount of time partially rewriting the Mars Rovers (source is here). I added a Swing UI so the rovers can be visually tracked.
Run it from command line
You can run it from the command line with
$ lein run
You can also overwrite the default number of rovers and the number of actions each can take before "freezing".
$ lein run ":rover-number 1000 :action-number 9999999"
It's a bit inelegant to use quotes, but I haven't had time yet to implement a proper command line interface.
UI
As you can see, the rovers seem to move in discrete steps instead of a continuous flow. This is partly due to the fact that the Swing panel is to refresh every time a Rover sends a position changed message to the Plateau, which triggers a message sent to the displaying component. We are talking about ~10,000 messages per second, so I had to introduce some sampling. You can read about it in one of the previous posts.
Performance
The number of moves taken by the Rovers is printed out to the console at every 10,000th move. With the default 100 Rovers on my MacBook Pro (16 GB 1600 MHz DDR3, 2.3 GHz Intel Core i7) this number is 740,000 in 20 seconds, so 37,000 moves per second. This many messages are running through the core.async channels. I don't know how the Scala actors perform, but seems to be pretty impressive. Once I tried with 10,000 Rovers, but all the cores jumped up to 100% and the cooling system started up so vigorously that I saw it better to kill the process. The max it seems to be able to take is 1,000 Rovers, but the number of moves per second is much lower in that case then the one with a 100.
Tests
Painfully lacking...
Next steps...
will be some better documentation, marking dead Rovers with different colours, tests, etc...
Run it from command line
You can run it from the command line with
$ lein run
You can also overwrite the default number of rovers and the number of actions each can take before "freezing".
$ lein run ":rover-number 1000 :action-number 9999999"
It's a bit inelegant to use quotes, but I haven't had time yet to implement a proper command line interface.
UI
As you can see, the rovers seem to move in discrete steps instead of a continuous flow. This is partly due to the fact that the Swing panel is to refresh every time a Rover sends a position changed message to the Plateau, which triggers a message sent to the displaying component. We are talking about ~10,000 messages per second, so I had to introduce some sampling. You can read about it in one of the previous posts.
Performance
The number of moves taken by the Rovers is printed out to the console at every 10,000th move. With the default 100 Rovers on my MacBook Pro (16 GB 1600 MHz DDR3, 2.3 GHz Intel Core i7) this number is 740,000 in 20 seconds, so 37,000 moves per second. This many messages are running through the core.async channels. I don't know how the Scala actors perform, but seems to be pretty impressive. Once I tried with 10,000 Rovers, but all the cores jumped up to 100% and the cooling system started up so vigorously that I saw it better to kill the process. The max it seems to be able to take is 1,000 Rovers, but the number of moves per second is much lower in that case then the one with a 100.
Tests
Painfully lacking...
Next steps...
will be some better documentation, marking dead Rovers with different colours, tests, etc...
Thursday 16 October 2014
Closures are the poor man's objects
I've heard this a lot without giving too much thought about it. But the Circuit Breaker made me contemplate the idea a bit. So....creating a stateful object in Clojure is actually pretty easy. The simplest I can think of is a unique id generator. This would look something like this in java
In Clojure
Of course it's a boring example so I've come up with another, slightly more interesting one, a simplified vending machine.
An epitome of applied Single Responsibility Principle. Does only one thing and hides its internals completely. Of course I could leverage dynamic typing to overload the function and perform any kind of other logic - getting information of the content, changing the state, whatever. I could make a mess, but I would have to awkwardly work against the language to do it.
In Clojure
Of course it's a boring example so I've come up with another, slightly more interesting one, a simplified vending machine.
An epitome of applied Single Responsibility Principle. Does only one thing and hides its internals completely. Of course I could leverage dynamic typing to overload the function and perform any kind of other logic - getting information of the content, changing the state, whatever. I could make a mess, but I would have to awkwardly work against the language to do it.
Wednesday 15 October 2014
Immutable datastructures vs information hiding
One generally accepted OO-concept Rich Hickey challenges with Clojure is the wisdom of information hiding. In rich object models (one proposals of DDD for example) aggregates are encouraged to eschew exposing their internal structure.
As a simple example we can imagine we are to write a card game with multiple players. Any card game actually, the point is that the players have hole cards which they don't share with others. Not even with the rest of the code. This project implements a simplified Blackjack game what perfectly serves the purpose of this post. This is the class for the participating players.
The player doesn't expose her cards, which is fine for us. Now imagine we want to spice up our game with some fantasy elements. Some player can cast Spying Spells to peek into their opponents' hands. This is orthogonal to the existing functionality, so the code change preferably shouldn't touch the existing code. But it does, because we have to modify Player some way.
Thus the Player class now serves two independent features of the game.
In Clojure classes are not used, instead functions operate on associative information models (@Copyright Rich Hickey), namely maps, lists, vectors, sets. There is no information hiding, the player is represented something like this
The logic is decoupled from the data, thus we can avoid awkward things like creating a new View object, but because Clojure uses immutable data structure there is no harm exposing their details either. We can easily add new functionality too without touching existing code.
As a simple example we can imagine we are to write a card game with multiple players. Any card game actually, the point is that the players have hole cards which they don't share with others. Not even with the rest of the code. This project implements a simplified Blackjack game what perfectly serves the purpose of this post. This is the class for the participating players.
The player doesn't expose her cards, which is fine for us. Now imagine we want to spice up our game with some fantasy elements. Some player can cast Spying Spells to peek into their opponents' hands. This is orthogonal to the existing functionality, so the code change preferably shouldn't touch the existing code. But it does, because we have to modify Player some way.
Thus the Player class now serves two independent features of the game.
In Clojure classes are not used, instead functions operate on associative information models (@Copyright Rich Hickey), namely maps, lists, vectors, sets. There is no information hiding, the player is represented something like this
The logic is decoupled from the data, thus we can avoid awkward things like creating a new View object, but because Clojure uses immutable data structure there is no harm exposing their details either. We can easily add new functionality too without touching existing code.
Tuesday 14 October 2014
Dynamic typing - Sampler filter
This post is superficially related to the basic concept of the previous, but shows a very similar pattern to Circuit Breaker. I call it Sampler filter, but I'm pretty sure there is an established name in the field of electrical engineering.
By implementing a Swing UI of my Mars Rover project (source code is here) I recently faced the problem that if I run hundreds of rovers, then refreshing the panel after every rover position change is impossible. Each position change puts an event - containing the positions for all rovers - on the channel the displayer logic listens to, and the messages just kept piling up. It's a typical IO bottleneck. So I decided it's enough to display every 100th or 1000th of them. They would change faster than my eyes can process them anyway. So I took the original function performing the display - repaint! - and wrapped it in a sampler. Here is the code
By implementing a Swing UI of my Mars Rover project (source code is here) I recently faced the problem that if I run hundreds of rovers, then refreshing the panel after every rover position change is impossible. Each position change puts an event - containing the positions for all rovers - on the channel the displayer logic listens to, and the messages just kept piling up. It's a typical IO bottleneck. So I decided it's enough to display every 100th or 1000th of them. They would change faster than my eyes can process them anyway. So I took the original function performing the display - repaint! - and wrapped it in a sampler. Here is the code
Thursday 2 October 2014
Dynamic typing - Circuit breaker
Despite my fascination for Clojure I'm yet to be sold on dynamic typing. Frankly the most frustrating thing about Clojure for me is the constant struggle with the lack of static type check. What went wrong and where? There are no types to guide me nor auto-completion, and I feel I have to hold too much in my head instead of letting the IDE help me out so I can concentrate on the problem. I read quite often arguments like "dynamic typing gives you a much greater flexibility" and "the static type system is often in your way", but honestly I don't understand them. Probably because I never have done anything serious in a dynamically typed language. So I've embarked on a journey of deliberately looking for situations when dynamic typing clearly gives an advantage. Here I'd like to show the first (and so far the only) one I found. Before looking at the more complex task of writing a circuit breaker, I'd like to start with a simple one.
Now this logger is able to transform any function with the (Int, Int) => Int signature to a logging one. Which is cool, but not cool enough. For every signature it has to be reimplemented. In a dynamically typed language you can use the same logger function for all. But this logger is a quite boring example, so I'd like to show something that is really useful and practical, the...
...Circuit Breaker
Imagine you application has to call a lot of web services, but the systems on the other end are quite unreliable. Sometimes they respond, but they have the habit of being unavailable for short time windows. You wouldn't like to waste time and thread calling them the n-th time if they haven't been responsive for a while. Better to wait a bit, then try again later. This is where the Circuit Breaker can help.
The Clojure version is
An easy way to try it out is to find a function that is simple to call with arguments that will break it, and wrap it in the circuit-breaker. The arithmetic divison is the perfect candidate.
Circuit Breaker is a fascinating pattern for many reasons. First it shows a clean analog to physical engineering solutions, making software engineering look like real engineering, and also making it easy to understand. Second, from an ezoteric FP point of view, Circuit Breaker is like magical entity that swallows a "stupid" function and spits out its doppelganger, same in signature and behaviour, but with an extra "intelligence".
The reason why dynamic typing fits so well to circuit breaker, I think, is that the extra functionality it adds has nothing to do with what the function does, nor with its signature.
Logger wrapper
Imagine you'd like to log the call of some functions. In Scala or Clojure it's an easy thing to do with higher-order functions.
Now this logger is able to transform any function with the (Int, Int) => Int signature to a logging one. Which is cool, but not cool enough. For every signature it has to be reimplemented. In a dynamically typed language you can use the same logger function for all. But this logger is a quite boring example, so I'd like to show something that is really useful and practical, the...
...Circuit Breaker
Imagine you application has to call a lot of web services, but the systems on the other end are quite unreliable. Sometimes they respond, but they have the habit of being unavailable for short time windows. You wouldn't like to waste time and thread calling them the n-th time if they haven't been responsive for a while. Better to wait a bit, then try again later. This is where the Circuit Breaker can help.
The Clojure version is
An easy way to try it out is to find a function that is simple to call with arguments that will break it, and wrap it in the circuit-breaker. The arithmetic divison is the perfect candidate.
The reason why dynamic typing fits so well to circuit breaker, I think, is that the extra functionality it adds has nothing to do with what the function does, nor with its signature.
Tuesday 16 September 2014
State in Functional Programming and OO
Just some musing...
In OO the global state, present in the old procedural programming, is broken up and the shards are moved into small, independent moving parts of the system, the objects. The objects don't expose their state, instead they expose behaviour. So the state disappears and and intricate net of acting agents fill its place. I have, like OO-fans, always seen this as an advantage, fascinated by the elegance of the transformation itself. But the disadvantage became obvious when I started learning FP. Since the state is obfuscated behind the objects' APIs, for a developer it can be hard to determine in exactly what state the system is. When I call a method on an object to return something it might send a message to another object or write into the DB and I won't know about it. Debugging is hard. Again I'm not sure it's a bad thing and I suppose sometimes it's good, e.g. in the case of logging. FP goes against this information hiding paradigm and promotes side-effect-free programming. Of course a program should have side-effects, otherwise what's the point? But FP forces the programmer to be very explicit about side-effects (= state changes). In an ideal FP architecture the inside of the onion is composed of pure functions and all the "impure stuff", like DB-operations, user input reading, web-service calling, etc, are in the outer layer, very visible, very explicit. So paradoxically, FB, being a state-shunning paradigm, is much more about the state than OO, which is about behaviour.
When I have some time and feel the creativity to do it, I'll append some examples.
Tuesday 2 September 2014
Mars Rovers in Clojure - Rewriting an Akka project to a Core Async one
I've decided to to reimplement Kaloz's Mars Rover Scala (for brevity's sake SMR from now on) project in Clojure (CMR). SMR is based on Actors of the Akka framework. Actors are the manifestation of the original idea of object orientation, that is, independent entities hiding their internal state and communicating via message passing. Since the whole model is purely about side effects, it seemed a nice challenge to implement in a functional way.
I wanted to have a jar as an end product that can be substituted to the backend of the original Scala, using the original presentation layer. I'm not there yet, but a first version is available on Github, where mars rovers can run in the REPL, spitting out their positions simply to the console.
All in all the project is a nice round 600 LOC. The Scala version had 548, which is not surprising. Scala, much like Clojure, is a very concise language, but because of its syntax you can compress the code more and still retain readability.
The main differences between SMR and CMR are
I wanted to have a jar as an end product that can be substituted to the backend of the original Scala, using the original presentation layer. I'm not there yet, but a first version is available on Github, where mars rovers can run in the REPL, spitting out their positions simply to the console.
All in all the project is a nice round 600 LOC. The Scala version had 548, which is not surprising. Scala, much like Clojure, is a very concise language, but because of its syntax you can compress the code more and still retain readability.
The main differences between SMR and CMR are
- Instead of complecting state, identity and behavioural logic, the Clojure version uses pure functions to capture the behaviour.
- All the "actors" have a receive method that takes the state of the "actor" and the received message (plus some additional information sometimes) and returns the new state with possibly a list of messages and executable effects.
- Context changes in actors are implemented in a surprisingly elegant way via multimethods. Different version of receive is called based on the state of the "actor"
- The actors are simple atoms. The state of the atom is the input of the receive and the state part of the output is swap!-ed into it
- The Clojure version, for the time being, lacks the distributed nature of Akka, and also its error handling mechanism.
- It uses the core.async for asynchronous communication. Where in the Scala version actors hold references to other actors, here channels are passed around
- The architecture, not surprisingly, is quite different
Application = 75% pure functions + 6% glue + 8% application-level + 11% other
During the reimplementation I dissected the code to form 4 distinct parts. LOC-wise the sizes of these parts are
- 75% is pure, framework-neutral functional code. I moved all those source files under marsrover.pure package.
- 6% is the glue, a mini messaging-framework I built over core.async in marsrover.glue.clj. In this "framework" a component's state is preserved in an atom. The component has an in-channel, an atom, and a function defining its behaviour. When a message arrives on the channel the component's state (value of the atom) and the message are the inputs of the behaviour function, and the output is the new state, the messages it has to send out, and possibly some effects. It's all very high level, doesn't know anything about our particular domain.
- 8% is the application layer, it's in marsrover.app.clj. This takes the pure part and injects it in the glue.
- 11% is other stuff like reading the expedition config (number of rovers, size of the plateau, etc).
So there is a big bulk of pure functional blob and a very thin layer glueing it together to form a proper application. I find this really fascinating. In the original Scala/Akka implementation the Akka framework and the logic was undisentangibly intertwined. Now the glue uses core.async, but if I wanted to use an actor-framework instead or any other approach, I would only need to change that 35 lines confined in a single source file. I say it again, it's fascinating. With functional approach we can push the framework/technology -related code to the outmost layer of our application, making it seem almost insignificant. The pure functions won't change if we want to move from the asynchronous model to synchronous, or replace core.async with any other integration solution. Imagine you'd wanna rewrite Mars Rovers in Scala, but not using Actors.
Finally, you can run the app by running marsrover.mars_expedition.clj in the REPL.
Finally, you can run the app by running marsrover.mars_expedition.clj in the REPL.
Friday 15 August 2014
New module added to Principle
I've added a new module to JPrinciple recently to enable the developer to constrain access to third party libraries to designated parts of the code. The reason for this is that I sometimes see processing json in app layer or using the Mongo API from the domain in our codebase. Being an ardent DDD proponent, I just cannot have that happening again! So the idea is that you can specify which libraries you allow access to in which layer. All the layers around the specified one can use those libraries of course, but nothing under it. Hope it made some sense. Probably it's better to see through an example
The new part is the <thirdparty>...</thirdparty>. It says that anything under org.apache.commons can be referenced in the Domain layer and up (app and infrastrucure), and the Infrastructure layer can use anything under the listed package paths: org.apache.maven,org.json,org.yaml,com.google.common.collect,jdepend. Anything else used in Infrastructure, e.g. something under org.mongo, would cause an alert. So would the reference to org.json in Domain, since it's only allowed in Infrastructure.
The visual idea behind it is that your code is a castle with multiple circles of defending walls around it. You let foreign troops leaking into your territory only until certain walls. Different troops can have different privileges, one (org.yaml) can only set up his tent inside the outmost wall (Infrastructure), the other (org.apache.commons) is allowed to enter the inner sanctum (Domain). Usable from version 0.25.
The new part is the <thirdparty>...</thirdparty>. It says that anything under org.apache.commons can be referenced in the Domain layer and up (app and infrastrucure), and the Infrastructure layer can use anything under the listed package paths: org.apache.maven,org.json,org.yaml,com.google.common.collect,jdepend. Anything else used in Infrastructure, e.g. something under org.mongo, would cause an alert. So would the reference to org.json in Domain, since it's only allowed in Infrastructure.
The visual idea behind it is that your code is a castle with multiple circles of defending walls around it. You let foreign troops leaking into your territory only until certain walls. Different troops can have different privileges, one (org.yaml) can only set up his tent inside the outmost wall (Infrastructure), the other (org.apache.commons) is allowed to enter the inner sanctum (Domain). Usable from version 0.25.
Wednesday 13 August 2014
Clojure, syntax and readability
def isValidVoter(p:Person) = p.age > 18 && !p.isInPrison
vs
(defn is-valid-voter [p] (and (> (:age p) 18) (not (:in-prison p))))
The first expression is in Scala, the second in Clojure. As much as I appreciate the simpleness of Clojure-syntax, I can't help but find the Scala version much easier to read. The Clojure version can be enhanced by line breaks
(defn is-valid-voter [p]
(and
(> (:age p) 18)
(not (:in-prison p))))
but now it's a four-liner instead of a one-liner. And yet still less readable than its Scala counterpart. Furthermore, if I want to remove the "in-prison" part, in Scala I just remove the end of the expression from the && sign without touching anything from the left of it.
def isValidVoter(p:Person) = p.age > 18 && !p.isInPrison
Back to the question of readability, part of it is due to our learnt preference for infix notation. But the real reason is, I think, is that the different parts of the function definition, such as name, arguments, body, just don't stand out in Lisp. Check this
(defn wrap [x] [x])
It lacks arbitrary rules like "there is a '=' between the expression signature and the body", "prefix a boolean expression with an '!' negates it" or "get the field of an object with <object>.<field>", and so on. Taking uniformity to its extreme, undeniably some readability is lost. But what is gained is not only the esthetic statisfaction over the ultimate simplicity, but the power coming from homoiconity, macros. Macros are discussed elsewhere, but I'm as much as fascinated by the simplicity itself as by the power it brings. There are many syntaxes with different advantages and disadvantages. Languages are evolving and we can't know how a programming language will look like in a hundred years. But Lisp is on a very end of the spectrum. It cannot get any simpler.
Friday 8 August 2014
OO, invariants and Clojure
In an object-oriented language if we'd like to implement a "concept", we create a class. Concept is too abstract (and probably there are better words out there for what I tried to say), so let's see a practical example. I want to implement a game of cards and therefore I'd like to deal with "decks". My OO-language of choice is Scala. Assuming that the operations I want to do on a deck are pulling a card from top, shuffle, and adding cards on the top, the code would look like this: (ignore Scala's awkward way of simulating enumerations - why is that so bad in this otherwise beautifully concise language is beyond me).
I chose to implement the deck as an immutable objects, all functions on it creating a new instance and leaving the original intact. I could have done the more OO-way of mutable state, but it's irrelevant for the case and I'd like to move the solution close to the one of Clojure discussed later.
See the check in the "constructor" and the isValidDeck function. This ensures that our Deck is always "valid". All the code using this class can safely assume that it doesn't contain duplicate cards. It's the simplest invariant I could come up, but serves the purpose of the post well. If I try to get out of my default OO-mindset acquired in the last 10 years and look at it with fresh eyes, what I see that we have a "raw" data structure, a list, and certain constraints attached to it. The data structure and the constraints together are the class.
The state-space of the Deck object obviously couldn't be bigger than that of the list, the raw data. So building a class around a raw data structure is simply imposing restrictions on how the underlying data can be modified by the clients of the class. Of course the methods of a class can hide side-effects, too, but again, in the aspect of invariants, it's not relevant.
The benefit of the OO approach is that the Deck stands guard over its invariants, so the constrains and the data "travel together". The disadvantage is that we lose all the already existing data-manipulating functions the language would provide out of the box. Let's demonstrate it with the Clojure solution.
Delightfully simple. We didn't have to write any of the required functions, the language gives us them all and plenty more. What if we want to remove all the Clubs from the Deck? Or halve it? Or order by some specific rule? Clojure would offer a million different data manipulation choices. So would Scala anyway if you choose to use it that way, but the language philosophy prefers encapsulating data and behaviour in one unit (class).
The disadvantage of Clojure's lightweight way is that we now lost the guard on the invariants. You can pass an arbitrary list to any functions expecting a valid deck, opening the door for hard to detect bugs. However I think there is a remedy for this and quite an easy one of that.
Using the 'with-invariant' function we can apply any validation with any function operating on any data structure. We can extend it easily to deal with multiple inputs, too, or different validations for the input and the output. Object-oriented programming ties validation and data together, offering very strong safeguards, but constraining reusability immensely.
Imagine now we want to extend our game of cards program by allowing different kind of games, including ones where the deck can contain duplicates. Or requires different operations on the deck (like pull from the bottom). Can we reuse the Deck class? Not in this current form. Maybe subclassing it or extract some traits? Either way the code starts to get complicated.
With Clojure we just use the same basic data structure (list) a new validator function with with-invariant or build it up in one single function for brevity's sake. The Clojure implementation dissected the class and cleaved off the constraints from the data. The data structure can processed in almost every imaginable way with the rich set of functions built-in the language, and the constraints can be plugged-in the computation whenever needed.
I chose to implement the deck as an immutable objects, all functions on it creating a new instance and leaving the original intact. I could have done the more OO-way of mutable state, but it's irrelevant for the case and I'd like to move the solution close to the one of Clojure discussed later.
See the check in the "constructor" and the isValidDeck function. This ensures that our Deck is always "valid". All the code using this class can safely assume that it doesn't contain duplicate cards. It's the simplest invariant I could come up, but serves the purpose of the post well. If I try to get out of my default OO-mindset acquired in the last 10 years and look at it with fresh eyes, what I see that we have a "raw" data structure, a list, and certain constraints attached to it. The data structure and the constraints together are the class.
Class = raw data structure + constraints
The state-space of the Deck object obviously couldn't be bigger than that of the list, the raw data. So building a class around a raw data structure is simply imposing restrictions on how the underlying data can be modified by the clients of the class. Of course the methods of a class can hide side-effects, too, but again, in the aspect of invariants, it's not relevant.
The benefit of the OO approach is that the Deck stands guard over its invariants, so the constrains and the data "travel together". The disadvantage is that we lose all the already existing data-manipulating functions the language would provide out of the box. Let's demonstrate it with the Clojure solution.
Delightfully simple. We didn't have to write any of the required functions, the language gives us them all and plenty more. What if we want to remove all the Clubs from the Deck? Or halve it? Or order by some specific rule? Clojure would offer a million different data manipulation choices. So would Scala anyway if you choose to use it that way, but the language philosophy prefers encapsulating data and behaviour in one unit (class).
The disadvantage of Clojure's lightweight way is that we now lost the guard on the invariants. You can pass an arbitrary list to any functions expecting a valid deck, opening the door for hard to detect bugs. However I think there is a remedy for this and quite an easy one of that.
Using the 'with-invariant' function we can apply any validation with any function operating on any data structure. We can extend it easily to deal with multiple inputs, too, or different validations for the input and the output. Object-oriented programming ties validation and data together, offering very strong safeguards, but constraining reusability immensely.
Classes vs (raw data + validator functions) = safety vs reusability
Imagine now we want to extend our game of cards program by allowing different kind of games, including ones where the deck can contain duplicates. Or requires different operations on the deck (like pull from the bottom). Can we reuse the Deck class? Not in this current form. Maybe subclassing it or extract some traits? Either way the code starts to get complicated.
With Clojure we just use the same basic data structure (list) a new validator function with with-invariant or build it up in one single function for brevity's sake. The Clojure implementation dissected the class and cleaved off the constraints from the data. The data structure can processed in almost every imaginable way with the rich set of functions built-in the language, and the constraints can be plugged-in the computation whenever needed.
Thursday 17 July 2014
Implementing a DDD project in Clojure - 2
One important thing I forgot to mention (even realize) in the first post is the applicability of the Dependency Inversion Principle for Clojure. In a DDD application adhering to Hexagonal Architecture the Domain layer is at the bottom of the dependency-chain. It has interfaces (Secondary Ports) implemented (as Secondary Adapters) in the Infrastructure layer, so the direction of dependency is the inverse of the direction of control-flow. In Clojure you can achieve the same with Protocols, but without application context and objects we need a Namespace (I called it registry) to instantiate the implementations and the Namespaces that use the Protocol have to depend on it. So through the registry the high-level Namespaces depend on the low-level ones. I don't know yet how to resolve it, if it's doable at all.
Friday 11 July 2014
Clojure - " java.lang.Exception: Cyclic load dependency"
I've just discovered that Clojure, by default, doesn't allow cyclic dependencies between namespaces, as opposed to Java, where those between classes are allowed. Cyclic dependencies being the root of many evils, that is a very good thing. It would be nice to find the same strictness imposed on package levels, but it's not.
Implementing a DDD project in Clojure - 1
In the last couple of weeks I've been occupying myself with porting the Blackjack Java project to Clojure. I've been curious whether DDD concepts could be reused in a functional language without sacrificing idiomaticity. There is still a long way to go, but the walking (limping) skeleton is ready. Check it on github. In the followings I make an attempt to summarize my experiences.
1. Aggregates
First of all Clojure strongly encourages working with immutable data structures. My naive assumption had been that an OO aggregate can be simply substituted with a data structure accompanied by a handful of functions operating on it. This is only partially true. Though I've found state mutation easily replaceable with creating a new data structure (Clojure's persistent data structures make it performant, and its very effective data manipulation functions easy) as the output of the applied function, but I'm not convinced about how invariants should be preserved in the absence of an entity with mutable state. To some degree it can be achieved by :pre and :post constraints on the individual functions, but I haven't really dived into it.
2. Domain events
Related to the previous point, since in OO aggregate roots are to publish domain events. Initially I implemented my "aggregates" the same way, the functions operating on them published events as a side-effect. But eventually I've realized it's very non-FP, so instead of invoking a side-effect, I changed these functions to return not only the updated data structure (the new state of the "aggregate"), but the event-list as well. Then the new "state" is updated in the DB and the events are sent out. This solution (which could be applied in OO as well) eliminates the need of thread-local/event store-flushing magic I resorted to in the java version. I am more than satisfied with this, it's not only idiomatic, but makes things so much simpler.
3. AOP for locking
This is quite pleasing also. Instead of aspects and annotations, it's a 5-(short)-line macro.
4. Layering
The most interesting part. In the very few books mentioning the large-scale structure of FP-programs I've stumbled across the notion of onion-layering, not unlike that of DDD or Hexagonal Architecture. We should strive to confine the mutable states to the outer layers, keeping the core completely functional. Unfortunately in OO-DDD, the "core" is the Domain, which does have logic with side-effects. Repository interfaces and Domain Services as front-doors to ACLs are very much part of the Domain, and are there to enable the Domain to interact with the external world (writing to DB, sending messages, calling web services, ...).
So quite early it became clear that the layering in FP-programs requires some changes in the OO-way of thinking.
4.1 Hexagonal Architecture
If we forget about DDD temporarily and reach back to HA (which I think is one of the foundations of the former), some parallels with the FP-version of onion-layering are obvious. In HA, the Adapters are outside of the core and the Ports are the joints that bind the Adapters to the core. Even an FP-program must interact with the external world (otherwise what's the point of writing it), so HA, with its explicitly designated areas of external interaction seems to be a natural fit.
4.2 Application layer
Having a clean facade for an application has nothing to do with OO or FP, so keeping this layer of DDD seemed ok. In addition to the normal DDD application layer responsibilities, such as locks, transactions and orchestration, I decided all calls to secondary (driving) ports should happen here (calls to a remote Bounded Context, Blackjack Wallet, interacting with the DB). This is the external layer of the onion (still beneath the adapters-layer, though).
I've kept different namespaces in this layer for the different modules (Player, Table, Game) and used most method names for commands with bangs.
4.3 Core (Domain) layer
A purely functional layer. I've tried to simply port the java version's Domain here, minus the secondary (driving ports), e.g. the Game-, Player-, TableRepository or the WalletService, which I've decided to put under a port package on the same level as app and domain. Secondary ports neither defined (but in their own package) or used (but in app services, see previous point) here.
4.4. Infrastructure layer
Quite the same as in the java version. Adapters are implemented as Records of Protocols, see next point.
5. Using Protocols for secondary ports
I've tried to avoid OO-techniques as much as possible, but protocols simply seem to be the right way to implement ports.
6. Application Context and Dependency Injection
Although in the absence of objects FP doesn't require DI the same way as an OO program would, I've found the use of Protocols has similar needs to that of objects. The Protocol implementation has to be accessible to the function that uses it without explicit dependency on it. In a Java-Spring application the application context would do this wiring, as well as instantiating the proper implementation based on the content of a property file. Lacking any framework I created a dedicated namespace for it, called registry. All namespaces where functions need to call Protocols import the registry namespace. I don't know how others do this, but this solution seemed straightforward.
7. Modules
I kept the module structure of the Java version intact. Game, Table, Player and Cashier are the same, they interact with each other in an asynchronous way. Clojure Async does this job perfectly. There is a channel for Domain Events and subscriptions for the event handlers. Much shorter and cleaner than the Java version's was.
To be continued...
1. Aggregates
First of all Clojure strongly encourages working with immutable data structures. My naive assumption had been that an OO aggregate can be simply substituted with a data structure accompanied by a handful of functions operating on it. This is only partially true. Though I've found state mutation easily replaceable with creating a new data structure (Clojure's persistent data structures make it performant, and its very effective data manipulation functions easy) as the output of the applied function, but I'm not convinced about how invariants should be preserved in the absence of an entity with mutable state. To some degree it can be achieved by :pre and :post constraints on the individual functions, but I haven't really dived into it.
2. Domain events
Related to the previous point, since in OO aggregate roots are to publish domain events. Initially I implemented my "aggregates" the same way, the functions operating on them published events as a side-effect. But eventually I've realized it's very non-FP, so instead of invoking a side-effect, I changed these functions to return not only the updated data structure (the new state of the "aggregate"), but the event-list as well. Then the new "state" is updated in the DB and the events are sent out. This solution (which could be applied in OO as well) eliminates the need of thread-local/event store-flushing magic I resorted to in the java version. I am more than satisfied with this, it's not only idiomatic, but makes things so much simpler.
3. AOP for locking
This is quite pleasing also. Instead of aspects and annotations, it's a 5-(short)-line macro.
4. Layering
The most interesting part. In the very few books mentioning the large-scale structure of FP-programs I've stumbled across the notion of onion-layering, not unlike that of DDD or Hexagonal Architecture. We should strive to confine the mutable states to the outer layers, keeping the core completely functional. Unfortunately in OO-DDD, the "core" is the Domain, which does have logic with side-effects. Repository interfaces and Domain Services as front-doors to ACLs are very much part of the Domain, and are there to enable the Domain to interact with the external world (writing to DB, sending messages, calling web services, ...).
So quite early it became clear that the layering in FP-programs requires some changes in the OO-way of thinking.
4.1 Hexagonal Architecture
If we forget about DDD temporarily and reach back to HA (which I think is one of the foundations of the former), some parallels with the FP-version of onion-layering are obvious. In HA, the Adapters are outside of the core and the Ports are the joints that bind the Adapters to the core. Even an FP-program must interact with the external world (otherwise what's the point of writing it), so HA, with its explicitly designated areas of external interaction seems to be a natural fit.
4.2 Application layer
Having a clean facade for an application has nothing to do with OO or FP, so keeping this layer of DDD seemed ok. In addition to the normal DDD application layer responsibilities, such as locks, transactions and orchestration, I decided all calls to secondary (driving) ports should happen here (calls to a remote Bounded Context, Blackjack Wallet, interacting with the DB). This is the external layer of the onion (still beneath the adapters-layer, though).
I've kept different namespaces in this layer for the different modules (Player, Table, Game) and used most method names for commands with bangs.
4.3 Core (Domain) layer
A purely functional layer. I've tried to simply port the java version's Domain here, minus the secondary (driving ports), e.g. the Game-, Player-, TableRepository or the WalletService, which I've decided to put under a port package on the same level as app and domain. Secondary ports neither defined (but in their own package) or used (but in app services, see previous point) here.
4.4. Infrastructure layer
Quite the same as in the java version. Adapters are implemented as Records of Protocols, see next point.
5. Using Protocols for secondary ports
I've tried to avoid OO-techniques as much as possible, but protocols simply seem to be the right way to implement ports.
6. Application Context and Dependency Injection
Although in the absence of objects FP doesn't require DI the same way as an OO program would, I've found the use of Protocols has similar needs to that of objects. The Protocol implementation has to be accessible to the function that uses it without explicit dependency on it. In a Java-Spring application the application context would do this wiring, as well as instantiating the proper implementation based on the content of a property file. Lacking any framework I created a dedicated namespace for it, called registry. All namespaces where functions need to call Protocols import the registry namespace. I don't know how others do this, but this solution seemed straightforward.
7. Modules
I kept the module structure of the Java version intact. Game, Table, Player and Cashier are the same, they interact with each other in an asynchronous way. Clojure Async does this job perfectly. There is a channel for Domain Events and subscriptions for the event handlers. Much shorter and cleaner than the Java version's was.
To be continued...
Friday 16 May 2014
Clojure practice - 2
After finishing the previous post I noticed that Kaloz's solution was a bit more concise, not leaning on a helper function. So here is a "more compact" version, even puttin the number->characters mapping definition in the same expression.
or, purely for the sake of example, having even the println in it
(defn compact-combinations [number]
(let [num->chars { \2 "ABC" \3 "DEF" \4 "GHI" \5 "JKL"
\6 "MNO" \7 "PQRS" \8 "TUV" \9 "WXYZ"}
red-fn (fn [acc d]
(m/match (num->chars d)
nil acc
chs (for [c chs
i acc]
(str i c))))]
(reduce red-fn [""] number)))
(println (compact-combinations "34"))
or, purely for the sake of example, having even the println in it
(let [combinations (fn [number]
(let [num->chars {\2 "ABC" \3 "DEF" \4 "GHI" \5 "JKL"
\6 "MNO" \7 "PQRS" \8 "TUV" \9 "WXYZ"}
red-fn (fn [acc d]
(m/match (num->chars d)
nil acc
chs (for [c chs
i acc]
(str i c))))]
(reduce red-fn [""] number)))]
(println (combinations "514")))
Thursday 15 May 2014
Clojure practice
Recently I have dabbled into Clojure. I've been planning a post for a while, but since now Kaloz has posted a Scala functional programming demonstration, the time has come to reply in kind. Please, read his post to get the problem, here I take the lazy way and simply show a Clojure solution.
(ns clojure-study.garbage
(:require [clojure.core.match :as m]))
(def num->chars {
\2 "ABC"
\3 "DEF"
\4 "GHI"
\5 "JKL"
\6 "MNO"
\7 "PQRS"
\8 "TUV"
\9 "WXYZ"})
(defn- red-fn [resolvedStrings number]
(m/match (num->chars number)
nil resolvedStrings
item (for [c item
i resolvedStrings]
(str i c))))
(defn combinations [number]
(reduce red-fn [""] number))
(println (combinations "77501998489"))
Thursday 1 May 2014
DDD reference project
I haven't posted in a while, but wasn't completely idle. I'd like to improve our way of doing DDD in the team I'm working in, so I have endeavoured to create a reference DDD project, where good practices, distilled experiences and some new ideas are demonstrated. The result is here, general explanatory notes on the wiki. It was quite a task, but a rewarding one.
Wednesday 26 February 2014
Defining your own exception types
I'm in a habit of creating different classes for different exceptions, as opposed to for example using RuntimeException-s or IllegalStateException-s describing details as a string in the constructor. I'm not hundred percent sure that's an ultimately better practice and recently I've watched a talk where the speaker adviced against class proliferation. True, in a language like Java, creating classes is very boilerplate. In Scala an exception class is probably half a line of code, and you can stuff a dozen in the same source file.
But, yesterday I realized there is a situation where a custom exception class comes handy. Imagine you put some validation logic in your domain class.
The Validate can be a validator class from Apache Commons or some custom class you wrote throwing IllegalArgumentException-s. At the project I'm working on right now we are using messaging and ActiveMQ as the integration solution. If there is an exception, the message is bounced back to the queue and the broker retries to deliver it a couple of times before it gives up and puts the message in the Dead Letter Queue. So if the message had a null as a player id, no matter how many times the broker retries it will always fail, but in the meantime the process consumes time and threads, possibly generating the same error log for a number of times. Instead of this if you create a( possibly abstract) n custom exception for domain problems, like
And at the entry point of your system you do a try-catch for it
Then you can catch and handle the problems where there is no chance of recovery in retrying.
But, yesterday I realized there is a situation where a custom exception class comes handy. Imagine you put some validation logic in your domain class.
public class PlayerId {
private final String id;
public PlayerId(String id) {
Validate.notNull(id);
this.id = id;
}
}
The Validate can be a validator class from Apache Commons or some custom class you wrote throwing IllegalArgumentException-s. At the project I'm working on right now we are using messaging and ActiveMQ as the integration solution. If there is an exception, the message is bounced back to the queue and the broker retries to deliver it a couple of times before it gives up and puts the message in the Dead Letter Queue. So if the message had a null as a player id, no matter how many times the broker retries it will always fail, but in the meantime the process consumes time and threads, possibly generating the same error log for a number of times. Instead of this if you create a( possibly abstract) n custom exception for domain problems, like
public abstract class DomainException extends RuntimeException {
public DomainException(String msg) {
super(msg);
}
}
And at the entry point of your system you do a try-catch for it
try {
// do the real thing
} catch(DomainException ex) {
//do something, e.g. logging
}
Then you can catch and handle the problems where there is no chance of recovery in retrying.
Tuesday 25 February 2014
JPrinciple
Finally, I've finished porting the code to Scala + made a wikipage on Github. By the way, Github is awesome.
So the source code: https://github.com/matemagyari/principle
And the Wiki: https://github.com/matemagyari/principle/wiki/JPrinciple
Tuesday 18 February 2014
DDD - Modules, Subdomains, Bounded Contexts
The exact meanings of these concepts have always eluded me, until yesterday I read some chapters from Implementing Domain-Driven Design, which has shed some light on the topic. I'd like to share some thoughts I think I've learnt.
Subdomains
Before going to discuss subdomains, let's talk about what Domain is in general. Domain is not something we build, but something that actually exists in the world. We only try to create a Model for it. Subdomains are also something existing independently of how they are modelled (if modelled at all). If we take Amazon as an example, users can browse the purchasable items, search for specific ones, have recommendations from Amazon, etc. This is a subdomain of the whole Amazon domain, we can call it Item Catalog subdomain for now. The user can also order items, view her previous order history, and manage her orders in some ways. This is the Order subdomain. These subdomains exist regardless of how Amazon implemented its application. Even if it's one monolithic ball of mud, using a single database table for everything, these subdomains are there. One purpose of DDD is to make them explicit and visible by a code base that reflects the structure of the Domain.
Bounded Contexts
As opposed to Subdomains, BCs are defined by the developers. A BC is where a ubiquitous language applies. Classes with the same name can mean different things in different BCs in the same domain. For example User in Item Catalog is someone with a browsing history, search patterns, recommendations. In the Order subdomain User is someone associated with orders and the ordering process. They are two different aspects of the same human being. But there is a one-to-one mapping between the two, probably manifesting in the same UserId.
Ideally, there is a one-to-one mapping between a Bounded Context and a Subdomain, so once the developers identified a subdomain, they design its own Bounded Context. Unfortunately in real world applications, where the boundaries of subdomains are not well recognised, BCs and Subdomains often overlap, a BC containing one or more Subdomains, or a single Subdomain is modelled with multiple BCs, or (in the messiest case) the overlap can be only partial.
I think a BC should be one deployable artifact (e.g. a jar or war file), but Vaughn mentions BCs implemented by multiple deployables, and I'm not sure whether packing multiple BCs in one component is a bad or good idea.
Modules
Modules in Java are simply packages. A Module can contain multiple aggregates if they are cohesive enough, or just one. Modules are really a less coarse-grained, lightweight versions of Bounded Contexts.
Subdomains
Before going to discuss subdomains, let's talk about what Domain is in general. Domain is not something we build, but something that actually exists in the world. We only try to create a Model for it. Subdomains are also something existing independently of how they are modelled (if modelled at all). If we take Amazon as an example, users can browse the purchasable items, search for specific ones, have recommendations from Amazon, etc. This is a subdomain of the whole Amazon domain, we can call it Item Catalog subdomain for now. The user can also order items, view her previous order history, and manage her orders in some ways. This is the Order subdomain. These subdomains exist regardless of how Amazon implemented its application. Even if it's one monolithic ball of mud, using a single database table for everything, these subdomains are there. One purpose of DDD is to make them explicit and visible by a code base that reflects the structure of the Domain.
Bounded Contexts
As opposed to Subdomains, BCs are defined by the developers. A BC is where a ubiquitous language applies. Classes with the same name can mean different things in different BCs in the same domain. For example User in Item Catalog is someone with a browsing history, search patterns, recommendations. In the Order subdomain User is someone associated with orders and the ordering process. They are two different aspects of the same human being. But there is a one-to-one mapping between the two, probably manifesting in the same UserId.
Ideally, there is a one-to-one mapping between a Bounded Context and a Subdomain, so once the developers identified a subdomain, they design its own Bounded Context. Unfortunately in real world applications, where the boundaries of subdomains are not well recognised, BCs and Subdomains often overlap, a BC containing one or more Subdomains, or a single Subdomain is modelled with multiple BCs, or (in the messiest case) the overlap can be only partial.
I think a BC should be one deployable artifact (e.g. a jar or war file), but Vaughn mentions BCs implemented by multiple deployables, and I'm not sure whether packing multiple BCs in one component is a bad or good idea.
Modules
Modules in Java are simply packages. A Module can contain multiple aggregates if they are cohesive enough, or just one. Modules are really a less coarse-grained, lightweight versions of Bounded Contexts.
Subscribe to:
Posts
(
Atom
)