Handling huge amounts of data

[Kent Boortz]

"Vlad Dumitrescu" <vlad_dumitrescu@REDACTED> writes:
> I am writing some programs that need to shuffle a lot of data (arrays/lists
> of some 5.000.000 items). I didn't want to use C/C++, so I settled to C#.
> None of the above, however, are especially good prototyping.
> 
> I would have loved to be able to give Erlang a chance, but some simple tests
> say most of the time goes to garbage collection (and I need some speed when
> running too, not only when developing). Also, the memory usage seems to
> always go up (except when a process ends and that memory is freed)[*].
> Besides that, it's occasionally causing "Abnormal termination"s.
> 
> Is there some "magic" that will make Erlang work with this kind of problems,
> or is it just not the right niche?

If you do message passing with huge terms you could try to compile an
Erlang/OTP that uses a shared heap to avoid the copying of the
data. You have to build Erlang/OTP from sources and configure like

  % ./configure --enable-shared-heap

I don't know if the shared heap support is stable enough or if the
garbage collector handle the case well but I think others on this list
can fill you in on that,

kent