Discrepancy between OS memory and erlang virtual memory?

[Jesper Louis Andersen]

Two shots in the dark:

// Your graph shows a memory spike close to the left area

One possibility is that you have some incoming data, which occurs rarely.
And when the processing of said data happens, you shoot through your 8Gb
limit. If this is the case, then it's often that you have smaller spikes as
well in the code. Your tools for dealing with this is
erlang:system_monitor/2, which allows you to monitor if the heap of a
process becomes very large. And also the process flag max_heap_size. It
allows you to narrow down where you are using excessive amounts of memory
and rewrite that part of the program to avoid using as much memory.

// You have lots of binary_alloc data

This is possibly because you have sub-binaries which are keeping the
underlying binary alive. If the underlying binary is large, and the
sub-binaries small, you can get into trouble. Nowadays, the VM is better at
handling these problems, and you might want to upgrade the VM from OTP21,
though I can't remember which older version improved the binary handling.
Your active mitigation tool in this case is binary:copy/1, which can force
a subbinary to be reallocated, freeing up the need to thread a 1-4megabyte
block around in the system. It's especially important if you have certain
binary keys in the system, which are large enough that they point to the
underlying binary.

The key idea is to figure out where the memory usage is, and why it's
spiking. Then think about a solution. One possible solution is to outright
reject incoming data that's too big, forcing the client to split work into
smaller units. Another possible solution is to look at streaming data flow,
rather than large up-front packets.


On Thu, Sep 23, 2021 at 8:13 AM Hitesh Kishorbhai Vaghani <
hitesh.v@REDACTED> wrote:

> Hi,
>
> We are eCommerce company and we use Erlang heavy in our production system.
> We are facing one issue which we are not able to figure out.
> My Erlang virtual machine shows 2.5GB memory usage but OS occupies 8GB
> plus RAM and the application getting restarted from time to time after RAM
> exhaust. We tried checking remote debugging no process is utilizing more
> memory in the erlang application. Process memory is also showing 2.56 GB.
> We are using ubuntu 20.04 and OTP 21.
>
> VM
> [image: VM.png]
>
> Service Status
> [image: Service status.png]
>
> OS Memory.png
> [image: OS Memory.png]
>
> Erlang Memory
> [image: erlang_memory_snap.png]
>
> Also Attaching Pmap screenshot.
>
> Is there anything we should be checking? How can we fix it? Any direction
> will help.
>
> *Regards,*
>
> *Hitesh Vaghani* | Engineering
>
> *E-Mail *|  hitesh.v@ <email@REDACTED>redbus.com | +91 9558476850
>
>
>
>
> *::DISCLAIMER::----------------------------------------------------------------------------------------------------------------------------------------------------The
> contents of this e-mail and any attachments are confidential and intended
> for the named recipient(s) only.E-mail transmission is not guaranteed to be
> secure or error-free as information could be intercepted, corrupted,lost,
> destroyed, arrive late or incomplete, or may contain viruses in
> transmission. The e mail and its contents(with or without referred errors)
> shall therefore not attach any liability on the originator or redBus.com.
> Views or opinions, if any, presented in this email are solely those of the
> author and may not necessarily reflect the views or opinions of redBus.com.
> Any form of reproduction, dissemination, copying, disclosure,
> modification,distribution and / or publication of this message without the
> prior written consent of authorized representative of redbus.
> <http://redbus.in/>com is strictly prohibited. If you have received this
> email in error please delete it and notify the sender immediately.Before
> opening any email and/or attachments, please check them for viruses and
> other defects.*



-- 
J.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20210923/4a666424/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: VM.png
Type: image/png
Size: 64989 bytes
Desc: not available
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20210923/4a666424/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Service status.png
Type: image/png
Size: 60256 bytes
Desc: not available
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20210923/4a666424/attachment-0005.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OS Memory.png
Type: image/png
Size: 8870 bytes
Desc: not available
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20210923/4a666424/attachment-0006.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: erlang_memory_snap.png
Type: image/png
Size: 16276 bytes
Desc: not available
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20210923/4a666424/attachment-0007.png>