[erlang-questions] [ann] erl-pipes: Hartmann pipelines implementation
Fri Aug 8 11:37:27 CEST 2008
I managed to put together an implementation of Hartmann pipelines that are
an extension to the pipes usually used in your OS of choice (see
http://en.wikipedia.org/wiki/Hartmann_pipeline for more details).
The code and some documentation (some of it already outdated) is to be found
Pipelines are formed by pipe segments connected to each other, forming a
directed graph. Data flows through the pipeline and is transformed along the
A pipe segment is represented by a process and messages are bearers of the
data flow. This allows multi-processor systems to split the load between the
There is a soft flow control mechanism (meaning that the control signals are
not authoritative) that prevents mailboxes to become overflowed.
A few pipe segments are defined in pips_builtins. One of the more
interesting ones is 'partition' that splits the flow in two, based on a
See pipes_test.erl for some examples and a simple benchmark.
This brings up the question of performance. Well, since I'm the author, I
will be nice and say there's a lot of room for improvement. I've measured
worst case runs that were 30x slower than doing this in the old-fashioned
way, but I've also had cases where the slowdown factor was 1.4, which is not
too bad. I suppose this kind of system would benefit a lot from a shared
For this to be easy to use, we will need a simple way to put together the
segments. A language to describe the graph that is "input friendly".
This is (I think) also a good example where channels might prove useful.
All in all, this is just a scratch on the surface. There is a lot to improve
and to add.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the erlang-questions