High Erlang memory binary

[Alin Popa]

Hi all,

I've stumbled upon a weird situation (you'll see down below what I mean by
weird) where my `system` memory is being quite high, out of which the
majority is assigned to `binary`.
When I do `:erlang.memory()`, I'm getting back the following values:
```
[
  total: 1012506552,
  processes: 84512344,
  processes_used: 83993696,
  system: 927994208,
  atom: 1845473,
  atom_used: 1816962,
  binary: 834439728,
  code: 48673550,
  ets: 11823024
]
```
As you can see, the binary is ~ 89% of the system memory (according to the
Erlang doc:
"binary
The total amount of memory currently allocated for binaries. This memory is
part of the memory presented as system memory.")

I wrote a function that gives me the memory taken by all processes refc
binaries, that looks like this:
```
total = fn ->
  :erlang.processes()
  |> Enum.map(fn pid ->
    try do
      {_, bins} = :erlang.process_info(pid, :binary)
      total_size = bins |> Enum.reduce(0, fn {_id, size, _refs}, acc ->
size + acc end)
      {pid, total_size, length(bins)}
    rescue
      _ -> {pid, 0, 0}
    end
  end)
  |> Enum.map(fn {pid, total_size, count} ->
    {pid, total_size, count}
  end)
  |> Enum.reduce(0, fn {_pid, total_size, _}, acc -> acc + total_size end)
end
```
The result returned by this function is in bytes 9600693, which is ~ 9 MB,
and not even near the values of `binary` returned by `:erlang.memory()`.

Now, the weird part is that my app is running inside a docker container,
and the docker container within a kubernetes cluster. In this setup I can
see the problem, but if I'm running it locally, within bare bones docker, I
can't see the same issue. And another thing I've noticed, even more
weirder, is that when the BEAM starts, the binary memory is ~ 300 MB, and
after each request it bumps up till it reaches ~ 900 MB, after which stays
constant - what I mean is that the BEAM doesn't crash with out of memory,
or something similar, but I'm still worried. Other applications I have,
also running on Erlang 22, and exactly the same docker image, don't suffer
from this problem, which makes me think this is related to the application
code...
This is also happening on both Erlang 21, and 22.

Another thing to mention is that I used as well `recon`, and after running
it, the binary section stays exactly the same, doesn't seem to be a matter
of not being garbage collected.

My questions are:
1. What other type of binary (beside refc process binaries) can live in
that zone of memory? Am I looking for the wrong thing?
2. How can I _look_ at the `binary` memory region to see what I have in
there?
3. Any other suggestion on what I might be doing wrong?

Any help will be much appreciated.

Thanks,
Alin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20200110/8c11b525/attachment.htm>