Handling huge amounts of data
Tue Jun 3 16:48:47 CEST 2003
we (AXD301) have had some success in reducing garbage collect cost by
increasing the heap size of data shuffling processes
from erlang:spawn_opt/4 doc;
Gives a minimum heap size in words. Setting this value
higher than the system default might speed up some
processes because less garbage collection is done.
Setting too high value, however, might waste memory
and slow down the system due to worse data locality.
Therefore, it is recommended to use this option only
for fine-tuning an application and to measure the exe-
cution time with various Size values.
> "Vlad Dumitrescu" <> writes:
> > I am writing some programs that need to shuffle a lot of data (arrays/lists
> > of some 5.000.000 items). I didn't want to use C/C++, so I settled to C#.
> > None of the above, however, are especially good prototyping.
> > I would have loved to be able to give Erlang a chance, but some simple tests
> > say most of the time goes to garbage collection (and I need some speed when
> > running too, not only when developing). Also, the memory usage seems to
> > always go up (except when a process ends and that memory is freed)[*].
> > Besides that, it's occasionally causing "Abnormal termination"s.
> > Is there some "magic" that will make Erlang work with this kind of problems,
> > or is it just not the right niche?
> If you do message passing with huge terms you could try to compile an
> Erlang/OTP that uses a shared heap to avoid the copying of the
> data. You have to build Erlang/OTP from sources and configure like
> % ./configure --enable-shared-heap
> I don't know if the shared heap support is stable enough or if the
> garbage collector handle the case well but I think others on this list
> can fill you in on that,
More information about the erlang-questions