Handling huge amounts of data
Tue Jun 3 20:18:31 CEST 2003
Thanks for the ideas.
> > If you do message passing with huge terms you could try to compile an
> > Erlang/OTP that uses a shared heap to avoid the copying of the
> > data. You have to build Erlang/OTP from sources and configure like
> Shared heap should be stable. If not, please send me the program causing
> the trouble.
I am on Windows, so rebuilding the emulator isn't an option... I will try to
set the minimum heap size.
My app has a huge list (4,5 milion items) of numeric data that can be
considered read-only and has to test it and filter it in different ways. I'd
like to keep this stable data in a separate process, so that the temporary
processing doesn't make that heap to be collected, but then there will be a
lot of copying between processes instead.
In the meanwhile, I found Lush (http://lush.sf.net) that is a simple Lisp
with special features for numeric processing. It might be something worth
More information about the erlang-questions