[erlang-questions] Memory-effective queue handling

Richard Carlsson carlsson.richard@REDACTED
Wed Jul 25 16:18:06 CEST 2012


On 07/25/2012 03:44 PM, Attila Rajmund Nohl wrote:
> 2012/7/25 Richard Carlsson <carlsson.richard@REDACTED>:
> [...]
>> You'd only run out of space if the queue is so long that you don't have
>> memory enough to call lists:reverse() on the current in-half of the queue
>> (and after that, the old queue will immediately become garbage). Can that
>> really be the case?
>
> It might happen. According to the crash dump the process in question uses
> Stack+heap: 228065700
>
> words of memory. I store a tuple of bigint and a small integer in the
> queue, and the queue is essentially a list (actually two lists, but
> doesn't matter), so one entry in the queue costs 2+3(?)+1+1=7 words.
> The process has some other stuff in its state, but that should be
> negligible. The erlang VM run for less than 13 hours. If I calculate
> correctly, this means that there should be around 600 events per
> second though 13 hours to reach this amount of memory. After some
> internal conversation I think it is possible that due to an unrelated
> bug I do get this much events.

Probably what you should use for a case like this is an ordered_set ets 
table, with the date as key and ets:last() to pick the oldest entry. The 
queue module is nondestructive and is nice in many ways, but it relies 
on amortized behaviour, and huge queues like yours make the intermittent 
periods of heavy internal work stick out like a sore thumb. The 
reversing of enormous lists isn't any good for responsiveness either, 
even if you had the memory. If you want to stick with a functional 
implementation, the gb_trees module will work pretty well, using 
gb_trees:smallest() to peek at the oldest entry.

    /Richard




More information about the erlang-questions mailing list