Today, when reactive programming is so popular concept and you can see ‘We’re reactive!’ claim on so many pages that have anything to do with scala (and not only),
Futures are something you have to deal all the time. Things like getting records from the database, be it good old sql, mongo, or elasticsearch, dispatching http requests, communication between actors - are all handled in asynchronous way and you’ll have to deal with
Futures whether you want it or not. You probably will want to, because they are as a matter of fact extremely useful and convenient concept. Async programing is a lot simpler thanks to them. You can easily wait on future completion, but as it’s discouraged practice in most cases, you would rather chain it or do something else with it before it comes back to you with the result. Quite often you have to deal with many futures at once. In such cases for comprehension comes handy,and because scala’s Future is actually a monad, you can do things like that:
In other words you can chain
Futures and perform some computations on their assumed results without waiting for them to complete. That’s neat and very useful. However, there’s a catch in the example above and that is that subsequent
Futures would be executed only after ones preceding them are complete. In other words you won’t have three parallel executions, just one after another. This makes sense however, and is what you’d expect and require in most cases. Just imagine sequence of calls to the database where every query relies on results obtained earlier. But there are also situations when resutls of
Futures are completely independent and there’s nothing that would justify running them one after another. That’d be just waste of time. Scala
Future has a nice and handy method,
sequence. Let’s see how we could use it:
We could also sum resutls right away like so:
That’s great, but more curious observer will notice that all this will work nice if
Futures forming the sequence are of the same type - in the example above it’s
Future[Int]. There are many situations where you have
Futures of different types and you’d like to execute them in parallel. Not a problem if the number is two:
Things get a bit hairy however when the number is three:
Yet it’s probably no challenge for you to create a method for handling three like so:
Yes… But something tells me you don’t want to take it one step further, to four, right? On the other hand, I bet you were in situations when it would be nice to have it, because otherwise it would make your scala code look more like clojure, when it would come to zip 5 or 6
Futures. Well, luckily, there’s a way to do it with any arbitrary number.
Before we’ll come up with the solution, let me tell a few words about shapeless. As the main page of the project states, shapeless is “a type class and dependent type based generic programming library for Scala”. That sounds a bit scary. Well, you need to bend your runtime mind a little bit in order to get end to end understanding of how does it work, but luckily we don’t have to take too deep dive here. Shapeless is about making attempt to solve things during compilation where they would normally be tackled in runtime. It’s rumored that Miles Sabin, the guy behind shapeless, doesn’t even exist at runtime. Shapeless aims to give you extreme level of confidence that if something compiles, it will work as expected. Let’s see some simple examples now.
Suppose you want to have a constraint in some function which would restrict lists passed in as an argument to be the same. Normally you would do something like this:
With shapeless though you can use
if you then call the function with following args:
it will work fine. However, this:
will not even compile. The reason is that type of
Sized("abc", "def") is
Sized[Seq[String], _2], but
Sized("ghi", "jkl", "mno") has type
Sized[Seq[String, _3]] which makes compiler not accept this code.
The most known part of shapeless, probably the one most widely used, is
HList, a heterogenous list. This is basically generalization of tuples and allows constructs like:
and then it allows operations that are not allowed on tuples, like
HLists be useful? Whenever you’d like to use tuples, but don’t know their sizes or miss some functionality on them. They have been found useful in spray routing classes, which every scala progammer is for sure familiar with.
Let’s try to mimic them using shapeless and then look closer at some aspects. First, some basic building blocks with obvious purpose:
Now, lets define our own simplified
Two lines that I’d like the reader to concentrate on first are
as they make use of very important and omnipresent concept of shapeless, that is providing implicit evidences for types and relationships between them.
implicit fp: FnToProduct.Aux[F, X => Route] we are providing evidence that function
f: F is convertible to
HList-like equivalent (the equivalent’s signature would be close to
def foo[X <: HList](x: X): Route).
With the other one,
implicit prepend: Prepend[X, Y], we are proving that two
HLists can be merged into third (or more precisely that there exists
HList type consisting of elements in which
We make use of these evidences and perform conversions making it possible to use regular functions along with hlist-like
happly. The fact that
Directives are parametrized with
HLists allows for great flexibility without sacrificing type safety.
So let’s define some concrete directives:
Now, let’s see how we could use them to construct routes:
Does it work? Yes!
Just look how easily we can combine two
Directives and make the combination work with
(String, String) => Route function, even though we have never defined
apply(f: (String, String) => Route) (or even
apply(f: String => Route)).
Now let’s try not to mimic anything but make something new. Let’s get back to our initial problem with futures. The intention must be clear now - we’d like to have a way of turning
HList. Something that would work like
Future.sequence, but would preserve the type of every compound and have no limitation on size. Let’s start with defining base trait:
So - we’d like to be able for given
HList to figure out it’s correspoing
IsHListOfFutures object and then have access to
Now, let’s see how that would look like for the simplest case, HNil:
Well, that’s kinda obvious, isn’t it? But now let’s take another step and try to tackle more complicated
We are here defining
IsHListOfFutures in terms of another
IsHListOfFutures, so doing a type-level recursion. At some point we’ll reach to to a implicit conversion from
HNilIsListOfFutures, and then, everything will be known about the type of what
hsequence has to return.
Now let’s put everything into
IsHListOfFutures object which we’ll be using for providing the evidences:
and define our
Once we have
IsHListOfFutures defined correctly and are able to provide it as an evidence, the implementation, as it turns out, is easy.
Let’s now look on specs I have prepared and see it in action:
So now, using this, you can combine even 22
Futures and perform one
hsequence call on them. Nice, isn’t it?
But some malcontents might ask is it really necessary to operate on
HLists, do we really have to construct these objects all the time we want to perform
hsequence operation? After all, it doesn’t look like “built into scala”. Let’s take it just one step further and see what can be done:
A few words on what we have just defined: we are providing two additional implicits to the
zip function - one,
gen, will be responsible to transforming function arguments into
HList. The other,
tupler, will be responsible for enabling doing the same, but in opposite direction and on results coming from
hsequence. Let’s see this in action:
Now that’s something that looks ‘native’, isn’t it?
And that’s that. Using shapeless we have managed to create nice function to deal with parallel execution of arbitrary number of
Futures. I hope you found this post interesting and useful. Thanks for your attention!
PS. There are other people’s attempts to provide hsequence method. Check out and compare: