[erlang-questions] How to see which processes used ets table?

[ok@REDACTED]

"Björn-Egil Dahlberg" <wallentin.dahlberg@REDACTED> wrote:

> We, well I, implemented a "limits" prototype where I could set max heap
> sizes on processes. The process would be killed with an exception limit
> error if it ever reached its max heap size. [snip]
>
> We ultimately decided against it, citing bad application design as exhibit
> A and root cause to runaway memory consumption; thus not strictly needed.
> =)

That's not why it isn't needed, that's why it IS needed.

> I think the biggest advantages of limits is like you say, catching
> problems early in design.

I'm writing this reply through a pain-in-the-posterior WebMail
system.  Why?  Because my normal machine is unavailable.  Why?
Because (a) my files were arbitrarily moved off my machine onto
NFS some years ago, *despite* my explicit request that my files
never be moved again without consulting me, (b) Mac OS X 10.6.8,
which has served me well with no problems, apparently got one
update too many (said to be to address Java problems) last week
and demanded to be shut down; (c) when it came back, NFS
connection was lost.  The technical support people say that the
simplest fix is to destroy my environment and give me 10.7
instead.  (I'm using 10.7 now, as a matter of fact, on a loaner
box; Mail does not recognise my old mailbox, my Dock set-up has
been trashed, and so it goes.)

The problem here is that a well designed and tested system that
has worked flawlessly for years can stop working as a result of
some apparently unrelated upgrade.

> However, I think you can simulate some of this already by using a server
> process with:
>
>     erlang:system_monitor(MonitorPid, [{large_heap, HeapSize}]),
>     ...
>     receive
>         {monitor, GcPid, large_heap, _} -> purge_with_fire(GcPid);
>     ...
>

What's wrong with this approach?

(1) It checks whether *ANY* process gets a heap exceeding the
    specified size.  But some processes might legitimately
    need a large heap, while others might be out of control if
    they grow to more than a few kilobytes.

(2) The limit that is checked is "the sum of the sizes of all
    memory blocks allocated for all heap generations".  But that
    is something that is entirely out of the control of an
    Erlang programmer.  The thing a programmer (or a process)
    can be held responsible for (within reason) is the number
    of words of memory actually *live*.

(3) *ANY* process can change this; it is not limited to the
    process itself (good for protection against accidents) or
    its creator (good for protection against malice).

(4) Judging from the description of system_monitor/0, there
    can only be ONE system monitor for ALL processes.  If
    there is a monitor watching for distribution problems
    (busy_dist_port) and an unrelated piece of code installs
    a monitor for large heaps, the distribution problem monitor
    is wiped away.

(5) As noted in the documentation, the monitoring process is not
    exempt from monitoring, so you can easily get into a
    recursive loop where memory overflow in the monitoring
    processes cause messages to be sent to it which cause ...

I could go on.  These are not the only flaws I see with this
interface.  Let's just say it's on my list of interfaces to
avoid while I still have my sanity.

The only alternative I've been able to think of with the
existing interfaces is to create a watchdog process that
checks process_info(Target, memory) periodically, which is
doable, and which can check for message queue growth as well.