[erlang-questions] Heap alloc error in lists:foldl/3

[Arif Ishaq]

Thanks for your answer. I understand that 16 such lists would occupy at least 16 * 80 MB = 480 MB, but I have a machine with 2 GB of physical RAM and as much again of virtual memory. I was monitoring the memory usage in the system and it didn't exceed 1.5 GB.

I suppose that the problem was nevertheless just that. Is there any way of "catching" the eheap_alloc in order to terminate things gracefully? 

Thanks again
Arif


-----Original Message-----
From: Ladislav Lenart [mailto:lenartlad@REDACTED] 
Sent: martedì 6 novembre 2012 15.30
To: Arif Ishaq
Cc: erlang-questions@REDACTED
Subject: Re: [erlang-questions] Heap alloc error in lists:foldl/3

Hello.

%% if I use sum, instead of foldsum, everything is ok

This is because foldsum/2 creates a list of 10 000 000 integers (call to lists:seq/2). And you create Np of such lists in total, each one in a different process. A list in Erlang is a chain of cons pairs (as in Lisp). Each cons occupies 2 words, where one word is either 32bits or 64bits, depending on your HW architecture. Thus one such list takes

    10 000 000 * 8 = 80MB (on 32-bit architecture)

Hence it is fairly possible that your system simply ran out of memory.

The version using sum/2 does not exhibit this behaviour, because it runs in constant space.


HTH,

Ladislav Lenart


On 6.11.2012 15:03, Arif Ishaq wrote:
> Hi,
>  
> I'm a newbie, playing with a toy programme to see how parallel 
> processing works in Erlang.
>  
> The goal is to calculate the value of pi using Leibnitz series. The 
> function
> pi/2 takes as arguments the number of processes, Np, and the number of 
> terms handled by each process, Nt.
>  
> The machine is an HP workstation with Windows Vista Enterprise.
>  
>  
> Here's the code:
>  
> -module(pp).
> -compile(export_all).
>  
> %% calculate PI using Np parallel processes, each calculating Nt terms 
> of the %% Liebnitz series
>  
> pi(Np, Nt) ->
>     Collector = spawn(?MODULE, collect, [Np, self()]),
>  
>     [spawn_opt(
>        fun() ->  Collector ! foldsum((N-1)*Nt, N*Nt -1) end      %% if I use
> sum, instead of foldsum, everything is ok
>       ) || N <- lists:seq(1,Np)],
>  
>     receive
>         Result ->
>             io:format("result is ~p~n", [4 * Result])
>     end.
>  
>  
> sum(Kfrom, Kto) ->
>     sum (Kfrom, Kto, 0).
>  
> sum(Kfrom, Kto, Sum) when Kfrom =:= Kto ->
>     Sum;
> sum(Kfrom, Kto, Sum) ->
>     sum(Kfrom+1, Kto, Sum + contribution(Kfrom)).
>  
>  
> foldsum(Kfrom, Kto) ->   
>     lists:foldl(
>       fun(K, Sum) -> Sum + contribution(K) end,
>       0,
>       lists:seq(Kfrom,Kto)).
>  
>  
> contribution(K) ->
>     Value = 1 / (2 * K + 1),
>     case K rem 2 of
>         0 ->
>             Value;
>         1 ->
>             -Value
>     end.
>  
>  
> collect (Np, For) ->
>     collect(Np, For, 0, 0).
>  
> collect (Np, For, Ncollected, Sum) when Np =:= Ncollected ->
>     For ! Sum;
> collect(Np, For, Ncollected, Sum) ->
>     receive
>         R ->
>             collect(Np, For, Ncollected + 1, Sum + R)
>     end.
>  
>  
>  
> If I try to calculate pi using 16 processes, each processing 1 million 
> terms, the result is ok.
> If, on the other hand, I try with 10 million terms each, erlang crashes.
> Even after closing the werl window, the process keeps running and has 
> to be killed brutally.
>  
>  
> Erlang R15B (erts-5.9) [smp:4:4] [async-threads:0]
>  
> Eshell V5.9  (abort with ^G)
> 1> c(pp).
> {ok,pp}
> 2> pp:pi(16,1000000).
> result is 3.1415925910897737
> ok
> 3> pp:pi(16,10000000).
>  
> Crash dump was written to: erl_crash.dump
> eheap_alloc: Cannot allocate 40121760 bytes of memory (of type "heap").
>  
>  
> Abnormal termination
>  
> If I try to read the crash dump with the crash dump viewer tool, I get 
> an error saying the file is not an erlang crash dump:
> erl_crash.dump is not an Erlang crash dump
>  
>  
> Why is this happening? Shouldn't foldl be tail recursive? Is this a 
> bug? Or am I doing something wrong?
>  
>  
> Thanks and best regards
> Arif Ishaq
>  
>  
> 
> 
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions