[erlang-questions] How to see which processes used ets table?

[Björn-Egil Dahlberg]

2013/2/5 Matthew Evans <mattevans123@REDACTED>

>  Hi,
>
> I don't wish to thread-jack, but I am anyway ;-)
>
> This pertains to a feature I would like to see in Erlang. The ability to
> set an optional "memory limit" when a process and ETS table is created (and
> maybe a global optional per-process limit when the VM is started). I've
> seen a few cases where, due to software bugs, a process size grows and
> grows; unfortunately as things stand today the result is your entire VM
> crashing - hopefully leaving you with a crash_dump.
>
> Having such a limit could cause the process to terminate (producing a OOM
> crash report in erlang.log) and the crashing process could be handled with
> supervisor rules. Even better you can envisage setting the limits
> artificially low during testing to catch these types of bugs early on.
>

Not this again.

This idea is not an uninteresting one. A safeguard against bad flow control
in application design.

We, well I, implemented a "limits" prototype where I could set max heap
sizes on processes. The process would be killed with an exception limit
error if it ever reached its max heap size. It worked fine in most cases
but if I remember correctly we couldn't guard against binary creation fast
enough (without killing performance). I'm pretty sure we could remedy those
problems though, but  by then my work queue was reprioritized.

We ultimately decided against it, citing bad application design as exhibit
A and root cause to runaway memory consumption; thus not strictly needed.
=)
I think the biggest advantages of limits is like you say, catching problems
early in design.
However, I think you can simulate some of this already by using a server
process with:

    erlang:system_monitor(MonitorPid, [{large_heap, HeapSize}]),
    ...
    receive
        {monitor, GcPid, large_heap, _} -> purge_with_fire(GcPid);
    ...

That might suffice for developing purposes.

If this should be considered again something more extensive then just
"limit heaps on processes and ets" should be considered I think. I would
like a subscriber system on erts where a could subscribe to different part
of the system and handle notifications as I see fit. The trouble here would
be "how fast do I need to handle this?". Do a message suffice or do I need
to wrap the notification in a fun like a callback?

Perhaps it has som validity to do something simple instead.

If a good enough proposal is put forward it will be considered.

// Björn-Egil


>
> Cheers
>
> Matt
>
> ------------------------------
> Date: Tue, 5 Feb 2013 08:51:16 +0100
> From: gustav.simonsson@REDACTED
> To: solomon.wzs@REDACTED
> CC: erlang-questions@REDACTED
> Subject: Re: [erlang-questions] How to see which processes used ets table?
>
>
> Note that erlang:memory/1 gives memory in bytes, whereas ets:i/0 gives it
> in words. On a 64-bit system 625M words would be 5GB, so the numbers you
> get might be correct. Try summing the memory from ets:i/0 up exact and
> compare it to erlang:memory(ets).
>
> Cheers,
> Gustav Simonsson
>
>
> On Tue, Feb 5, 2013 at 3:42 AM, Solomon <solomon.wzs@REDACTED> wrote:
>
> I checked system info with erlang:memory/1 and found the total amount of
> memory allocated for ets tables was about 5GB.
>
> Then I sum the memory size return by ets:i/0 and it was about 600MB.
>
> So which processes used ets table and did not return by ets:i/0?
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
>
>
> _______________________________________________ erlang-questions mailing
> list erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20130207/68b151b5/attachment.htm>