[erlang-questions] Heap alloc error in lists:foldl/3
Tue Nov 6 17:52:03 CET 2012
Thanks for your answer. I understand that 16 such lists would occupy at least 16 * 80 MB = 480 MB, but I have a machine with 2 GB of physical RAM and as much again of virtual memory. I was monitoring the memory usage in the system and it didn't exceed 1.5 GB.
I suppose that the problem was nevertheless just that. Is there any way of "catching" the eheap_alloc in order to terminate things gracefully?
From: Ladislav Lenart [mailto:]
Sent: martedì 6 novembre 2012 15.30
To: Arif Ishaq
Subject: Re: [erlang-questions] Heap alloc error in lists:foldl/3
%% if I use sum, instead of foldsum, everything is ok
This is because foldsum/2 creates a list of 10 000 000 integers (call to lists:seq/2). And you create Np of such lists in total, each one in a different process. A list in Erlang is a chain of cons pairs (as in Lisp). Each cons occupies 2 words, where one word is either 32bits or 64bits, depending on your HW architecture. Thus one such list takes
10 000 000 * 8 = 80MB (on 32-bit architecture)
Hence it is fairly possible that your system simply ran out of memory.
The version using sum/2 does not exhibit this behaviour, because it runs in constant space.
On 6.11.2012 15:03, Arif Ishaq wrote:
> I'm a newbie, playing with a toy programme to see how parallel
> processing works in Erlang.
> The goal is to calculate the value of pi using Leibnitz series. The
> pi/2 takes as arguments the number of processes, Np, and the number of
> terms handled by each process, Nt.
> The machine is an HP workstation with Windows Vista Enterprise.
> Here's the code:
> %% calculate PI using Np parallel processes, each calculating Nt terms
> of the %% Liebnitz series
> pi(Np, Nt) ->
> Collector = spawn(?MODULE, collect, [Np, self()]),
> fun() -> Collector ! foldsum((N-1)*Nt, N*Nt -1) end %% if I use
> sum, instead of foldsum, everything is ok
> ) || N <- lists:seq(1,Np)],
> Result ->
> io:format("result is ~p~n", [4 * Result])
> sum(Kfrom, Kto) ->
> sum (Kfrom, Kto, 0).
> sum(Kfrom, Kto, Sum) when Kfrom =:= Kto ->
> sum(Kfrom, Kto, Sum) ->
> sum(Kfrom+1, Kto, Sum + contribution(Kfrom)).
> foldsum(Kfrom, Kto) ->
> fun(K, Sum) -> Sum + contribution(K) end,
> contribution(K) ->
> Value = 1 / (2 * K + 1),
> case K rem 2 of
> 0 ->
> 1 ->
> collect (Np, For) ->
> collect(Np, For, 0, 0).
> collect (Np, For, Ncollected, Sum) when Np =:= Ncollected ->
> For ! Sum;
> collect(Np, For, Ncollected, Sum) ->
> R ->
> collect(Np, For, Ncollected + 1, Sum + R)
> If I try to calculate pi using 16 processes, each processing 1 million
> terms, the result is ok.
> If, on the other hand, I try with 10 million terms each, erlang crashes.
> Even after closing the werl window, the process keeps running and has
> to be killed brutally.
> Erlang R15B (erts-5.9) [smp:4:4] [async-threads:0]
> Eshell V5.9 (abort with ^G)
> 1> c(pp).
> 2> pp:pi(16,1000000).
> result is 3.1415925910897737
> 3> pp:pi(16,10000000).
> Crash dump was written to: erl_crash.dump
> eheap_alloc: Cannot allocate 40121760 bytes of memory (of type "heap").
> Abnormal termination
> If I try to read the crash dump with the crash dump viewer tool, I get
> an error saying the file is not an erlang crash dump:
> erl_crash.dump is not an Erlang crash dump
> Why is this happening? Shouldn't foldl be tail recursive? Is this a
> bug? Or am I doing something wrong?
> Thanks and best regards
> Arif Ishaq
> erlang-questions mailing list
More information about the erlang-questions