High Erlang memory binary

[Alin Popa]

Thanks so much, Lukas; I'll try to look at the inet ports for now, as I
already did some experimentation (i.e. killing some processes), and the
memory went down a bit (it goes back up, but this shows me that there may
be something around that area.

Alin


On Tue, Jan 14, 2020 at 7:55 AM Lukas Larsson <lukas@REDACTED> wrote:

>
>
> On Fri, Jan 10, 2020 at 8:04 PM Alin Popa <alin.popa@REDACTED> wrote:
>
>> Hi all,
>>
>
> Hello!
>
>
>>
>> I've stumbled upon a weird situation (you'll see down below what I mean
>> by weird) where my `system` memory is being quite high, out of which the
>> majority is assigned to `binary`.
>> When I do `:erlang.memory()`, I'm getting back the following values:
>> ```
>> [
>>   total: 1012506552,
>>   processes: 84512344,
>>   processes_used: 83993696,
>>   system: 927994208,
>>   atom: 1845473,
>>   atom_used: 1816962,
>>   binary: 834439728,
>>   code: 48673550,
>>   ets: 11823024
>> ]
>> ```
>> As you can see, the binary is ~ 89% of the system memory (according to
>> the Erlang doc:
>> "binary
>> The total amount of memory currently allocated for binaries. This memory
>> is part of the memory presented as system memory.")
>>
>> I wrote a function that gives me the memory taken by all processes refc
>> binaries, that looks like this:
>> ```
>> total = fn ->
>>   :erlang.processes()
>>   |> Enum.map(fn pid ->
>>     try do
>>       {_, bins} = :erlang.process_info(pid, :binary)
>>       total_size = bins |> Enum.reduce(0, fn {_id, size, _refs}, acc ->
>> size + acc end)
>>       {pid, total_size, length(bins)}
>>     rescue
>>       _ -> {pid, 0, 0}
>>     end
>>   end)
>>   |> Enum.map(fn {pid, total_size, count} ->
>>     {pid, total_size, count}
>>   end)
>>   |> Enum.reduce(0, fn {_pid, total_size, _}, acc -> acc + total_size end)
>> end
>> ```
>> The result returned by this function is in bytes 9600693, which is ~ 9
>> MB, and not even near the values of `binary` returned by `:erlang.memory()`.
>>
>> Now, the weird part is that my app is running inside a docker container,
>> and the docker container within a kubernetes cluster. In this setup I can
>> see the problem, but if I'm running it locally, within bare bones docker, I
>> can't see the same issue. And another thing I've noticed, even more
>> weirder, is that when the BEAM starts, the binary memory is ~ 300 MB, and
>> after each request it bumps up till it reaches ~ 900 MB, after which stays
>> constant - what I mean is that the BEAM doesn't crash with out of memory,
>> or something similar, but I'm still worried. Other applications I have,
>> also running on Erlang 22, and exactly the same docker image, don't suffer
>> from this problem, which makes me think this is related to the application
>> code...
>> This is also happening on both Erlang 21, and 22.
>>
>> Another thing to mention is that I used as well `recon`, and after
>> running it, the binary section stays exactly the same, doesn't seem to be a
>> matter of not being garbage collected.
>>
>> My questions are:
>> 1. What other type of binary (beside refc process binaries) can live in
>> that zone of memory? Am I looking for the wrong thing?
>>
>
> The main ones are binaries kept by ets tables and ports.
>
>
>> 2. How can I _look_ at the `binary` memory region to see what I have in
>> there?
>>
>
> You can use the instrument module in runtime_tools to get some more
> insights.
>
>
>> 3. Any other suggestion on what I might be doing wrong?
>>
>
> From what you describe above I would look at the inet ports that are
> running and make sure that they are closed and that they use sane buffer
> sizes.
>
>>
>> Any help will be much appreciated.
>>
>> Thanks,
>> Alin
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20200114/426b96f3/attachment.htm>