Developing a fast and durable pub/sub message bus in Haskell

Written by Will Sewell
Published: 2016-09-17 (last updated: 2016-10-12)

In this talk Will Sewell described Pusher's expierience with using Haskell for a realtime pub/sub message bus.

Slides: https://drive.google.com/open?id=0ByavZVi4q_BaUXdRa1ZRYWdkbjA

He started by describing the team's thoughts on Haskell after a year of use in an industrial setting:

Pros

  • Types
  • Concurrency
  • Tooling
  • Hiring

 Cons

  • Too many ways of solving the same problem
  • Error messages
  • Tooling is good, but can be fiddly to set up and there is unrealised potential

The remainder of the talk focused on a problem where GHC's GC had unacceptable pause times when the working set was large and long lived. This is generally because it is optimised for throughput over latency. See James Fisher's blog post and Stack Overflow question for more info. To solve this problem we would need a concurrent (like Go) or incremental GC (like Ocaml) for the old generation. Ultimately Pusher rewrote the system in Go.

There are future developments that could help:

  • The compacting collection algorithm could be worked on in order to get worst case latencies more in line with Ocaml or Go
  • Compact regions ([paper](http://ezyang.com/papers/ezyang15-cnf.pdf ))could be used so that long lived objects do not need to be traversed during a collection
  • An application of linear types could be a kind of type safe manual memory management if it is ever worked on

In conclusion Haskell is great, but think twice for realtime systems with a large working set!