Handling huge amounts of data

[Vlad Dumitrescu]

Thanks for the ideas.

Kent wrote:
> > If you do message passing with huge terms you could try to compile an
> > Erlang/OTP that uses a shared heap to avoid the copying of the
> > data. You have to build Erlang/OTP from sources and configure like

Jesper wrote:
> Shared heap should be stable. If not, please send me the program causing
> the trouble.

I am on Windows, so rebuilding the emulator isn't an option... I will try to
set the minimum heap size.

My app has a huge list (4,5 milion items) of numeric data that can be
considered read-only and has to test it and filter it in different ways. I'd
like to keep this stable data in a separate process, so that the temporary
processing doesn't make that heap to be collected, but then there will be a
lot of copying between processes instead.

In the meanwhile, I found Lush (http://lush.sf.net) that is a simple Lisp
with special features for numeric processing. It might be something worth
looking at.

regards,
Vlad