[erlang-questions] [ann] erl-pipes: Hartmann pipelines implementation

[Vlad Dumitrescu]

Hi,

I managed to put together an implementation of Hartmann pipelines that are
an extension to the pipes usually used in your OS of choice (see
http://en.wikipedia.org/wiki/Hartmann_pipeline for more details).

The code and some documentation (some of it already outdated) is to be found
at http://code.google.com/p/erl-pipes/

*Brief description:

Pipelines are formed by pipe segments connected to each other, forming a
directed graph. Data flows through the pipeline and is transformed along the
way.

A pipe segment is represented by a process and messages are bearers of the
data flow. This allows multi-processor systems to split the load between the
CPUs.

There is a soft flow control mechanism (meaning that the control signals are
not authoritative) that prevents mailboxes to become overflowed.

A few pipe segments are defined in pips_builtins. One of the more
interesting ones is 'partition' that splits the flow in two, based on a
user-supplied predicate.

See pipes_test.erl for some examples and a simple benchmark.
This brings up the question of performance. Well, since I'm the author, I
will be nice and say there's a lot of room for improvement. I've measured
worst case runs that were 30x slower than doing this in the old-fashioned
way, but I've also had cases where the slowdown factor was 1.4, which is not
too bad. I suppose this kind of system would benefit a lot from a shared
heap runtime.

For this to be easy to use, we will need a simple way to put together the
segments. A language to describe the graph that is "input friendly".

This is (I think) also a good example where channels might prove useful.

All in all, this is just a scratch on the surface. There is a lot to improve
and to add.

best regards,
Vlad
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20080808/af8d95a7/attachment.htm>