[erlang-questions] Question about potential application

[Bob Cowdery]

Rich
 
Thanks for responding. I have since discovered we need to cater for very large documents (100MB+) even though the majority will be very much smaller. I am considering a shared memory solution where the message is effectively in shared memory and a reference is passed so the next process in line can attach. If it needs to go across a machine boundary then its only one transfer. If anybody knows any reason why this might be problematical I would appreciate hearing about it.
 
Bob 

	-----Original Message----- 
	From: RCB [mailto:rcbeerman@REDACTED] 
	Sent: Wed 04/10/2006 16:55 
	To: Bob Cowdery 
	Cc: 
	Subject: Re: [erlang-questions] Question about potential application
	
	
	Bob,
	
	I wrote a commercial application in Erlang that does something similar.    Instead of using the ! operator to transport large quantities of data between nodes, I chose to use gen_tcp calls directly.  I did this mostly because I understood the performance characteristics of gen_tcp under heavy load.    It works very well, btw.   Let me know if you have specific implementation questions. 
	
	Rich
	
	
	
	On 10/4/06, Bob Cowdery <Bob.Cowdery@REDACTED> wrote: 

		Hi
		
		I am considereing Erlang as a bus to connect a pipeline process that performs certain work on documents that pass down the pipeline. There will be many stages in the pipeline all of which need to be scalable. Much of the real work will need to be carried out by existing C++ code at least in the near future. I can envisage a network of nodes that use C-drivers.Each node could instantiate multiple drivers as required and the node could be duplicated for scaling and resiliance. So far, so good. My question is about the data that passes through the workflow. The data is unstructured (documents) or semi-structured (XML at present) and can be large, lets say up to 100KB. Is it sensible to ship that data over Erlang end-to-end, does anybody do similar things. Other possibilities are to manage the data separately within or outside of Erlang and pass references only.I feel the first way is somewhat cleaner and without potential bottlenecks but the data size and transforms between the!
		  C and Erlang world worry me.
		
		Any thoughts appreciated.
		
		Bob
		
		
		_______________________________________________ 
		erlang-questions mailing list
		erlang-questions@REDACTED
		http://www.erlang.org/mailman/listinfo/erlang-questions 
		




	-- 
	Rich Beerman
	Cupertino, California
	mobile: 408/221-2444
	fax: 408/255-7944