Handling huge amounts of data

[Mats Cronqvist]

  we (AXD301) have had some success in reducing garbage collect cost by 
increasing the heap size of data shuffling processes
 from erlang:spawn_opt/4 doc;

       {min_heap_size, Size}:
           Gives a minimum heap size in words. Setting this value
           higher  than  the  system  default might speed up some
           processes because less  garbage  collection  is  done.
           Setting  too  high  value, however, might waste memory
           and slow down the system due to worse  data  locality.
           Therefore,  it  is recommended to use this option only
           for fine-tuning an application and to measure the exe-
           cution time with various Size values.

> "Vlad Dumitrescu" <vlad_dumitrescu@REDACTED> writes:
> > I am writing some programs that need to shuffle a lot of data (arrays/lists
> > of some 5.000.000 items). I didn't want to use C/C++, so I settled to C#.
> > None of the above, however, are especially good prototyping.
> > 
> > I would have loved to be able to give Erlang a chance, but some simple tests
> > say most of the time goes to garbage collection (and I need some speed when
> > running too, not only when developing). Also, the memory usage seems to
> > always go up (except when a process ends and that memory is freed)[*].
> > Besides that, it's occasionally causing "Abnormal termination"s.
> > 
> > Is there some "magic" that will make Erlang work with this kind of problems,
> > or is it just not the right niche?
> 
> If you do message passing with huge terms you could try to compile an
> Erlang/OTP that uses a shared heap to avoid the copying of the
> data. You have to build Erlang/OTP from sources and configure like
> 
>   % ./configure --enable-shared-heap
> 
> I don't know if the shared heap support is stable enough or if the
> garbage collector handle the case well but I think others on this list
> can fill you in on that,
> 
> kent