[erlang-questions] Question about potential application

[Bob Cowdery]

Hi

I am considereing Erlang as a bus to connect a pipeline process that performs certain work on documents that pass down the pipeline. There will be many stages in the pipeline all of which need to be scalable. Much of the real work will need to be carried out by existing C++ code at least in the near future. I can envisage a network of nodes that use C-drivers.Each node could instantiate multiple drivers as required and the node could be duplicated for scaling and resiliance. So far, so good. My question is about the data that passes through the workflow. The data is unstructured (documents) or semi-structured (XML at present) and can be large, lets say up to 100KB. Is it sensible to ship that data over Erlang end-to-end, does anybody do similar things. Other possibilities are to manage the data separately within or outside of Erlang and pass references only.I feel the first way is somewhat cleaner and without potential bottlenecks but the data size and transforms between the C and Erlang world worry me. 

Any thoughts appreciated.

Bob