[erlang-questions] How to see which processes used ets table?

[Björn-Egil Dahlberg]

2013/2/7 <ok@REDACTED>

> "Björn-Egil Dahlberg" <wallentin.dahlberg@REDACTED> wrote:
>
> > We, well I, implemented a "limits" prototype where I could set max heap
> > sizes on processes. The process would be killed with an exception limit
> > error if it ever reached its max heap size. [snip]
> >
> > We ultimately decided against it, citing bad application design as
> exhibit
> > A and root cause to runaway memory consumption; thus not strictly needed.
> > =)
>
> That's not why it isn't needed, that's why it IS needed.
>

The "=)" refers to the somewhat unconvincingly conclusion. Ironi is not
well suited for internet medium.
It's not without truth though. The flow control problem is solved via good
design, but I absolutely agree that limits are a handy tool for developers
and can certainly help developers find those issues.

Developers wants as much rope they can get from a system.

> I think the biggest advantages of limits is like you say, catching
> > problems early in design.
>
>
> The problem here is that a well designed and tested system that
> has worked flawlessly for years can stop working as a result of
> some apparently unrelated upgrade.
>

Well, normally one tests an upgraded system before deployment but I agree
with your point.


> > However, I think you can simulate some of this already by using a server
> > process with:
> >
> >     erlang:system_monitor(MonitorPid, [{large_heap, HeapSize}]),
> >     ...
> >     receive
> >         {monitor, GcPid, large_heap, _} -> purge_with_fire(GcPid);
> >     ...
> >
>
> What's wrong with this approach?
>

*Bracing for impact*


> (1) It checks whether *ANY* process gets a heap exceeding the
>     specified size.  But some processes might legitimately
>     need a large heap, while others might be out of control if
>     they grow to more than a few kilobytes.
>
> (2) The limit that is checked is "the sum of the sizes of all
>     memory blocks allocated for all heap generations".  But that
>     is something that is entirely out of the control of an
>     Erlang programmer.  The thing a programmer (or a process)
>     can be held responsible for (within reason) is the number
>     of words of memory actually *live*.
>
> (3) *ANY* process can change this; it is not limited to the
>     process itself (good for protection against accidents) or
>     its creator (good for protection against malice).
>
> (4) Judging from the description of system_monitor/0, there
>     can only be ONE system monitor for ALL processes.  If
>     there is a monitor watching for distribution problems
>     (busy_dist_port) and an unrelated piece of code installs
>     a monitor for large heaps, the distribution problem monitor
>     is wiped away.
>
> (5) As noted in the documentation, the monitoring process is not
>     exempt from monitoring, so you can easily get into a
>     recursive loop where memory overflow in the monitoring
>     processes cause messages to be sent to it which cause ...
>

This is great!

You have identified several problems that can be fed into a limits proposal
and how it should be solved.

Please note that system_monitor suggestion in this mail was intended as a
"fall-forward" (swedish idiom?). The development case .. but I think you
understood that.


I could go on.  These are not the only flaws I see with this
> interface.  Let's just say it's on my list of interfaces to
> avoid while I still have my sanity.
>

Haha. =) I will not be as judgmental in my opinion of that interface but it
has some shortcomings, that's for sure.


> The only alternative I've been able to think of with the
> existing interfaces is to create a watchdog process that
> checks process_info(Target, memory) periodically, which is
> doable, and which can check for message queue growth as well.
>

I wouldn't recommend such an approach either. Especially if you want it in
production. You will kill any sort of performance if you do this. Also, if
you implement something within erts that tries to check the *complete* msg
queue (both inner and outer) you will also kill any sort of application
performance.

I think the best approach is the garbage collector still. You have complete
access to the process and can make judgment calls there.

Also, processes is just one type of entity for limits. You have to consider
*all* aspects of the runtime system if you wish to make a limits proposal
that can stop OOM problems.

I'm still not convinced that this should be prioritized over something
else. Runaway memory is, most of the time, bad flow control. Wouldn't it be
better to solve that instead?

// Björn-Egil
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20130207/79ccf6d6/attachment.htm>