<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jan 10, 2020 at 8:04 PM Alin Popa <<a href="mailto:alin.popa@gmail.com">alin.popa@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi all,</div></blockquote><div><br></div><div>Hello!</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>I've stumbled upon a weird situation (you'll see down below what I mean by weird) where my `system` memory is being quite high, out of which the majority is assigned to `binary`.</div><div>When I do `:erlang.memory()`, I'm getting back the following values:</div><div>```</div><div>[<br> total: 1012506552,<br> processes: 84512344,<br> processes_used: 83993696,<br> system: 927994208,<br> atom: 1845473,<br> atom_used: 1816962,<br> binary: 834439728,<br> code: 48673550,<br> ets: 11823024<br>]<br></div><div>```</div><div>As you can see, the binary is ~ 89% of the system memory (according to the Erlang doc: </div><div>"binary</div>The total amount of memory currently allocated for binaries. This memory is part of the memory presented as system memory.")<br><br><div>I wrote a function that gives me the memory taken by all processes refc binaries, that looks like this:<br>```<br>total = fn -><br> :erlang.processes()<br> |> Enum.map(fn pid -><br> try do<br> {_, bins} = :erlang.process_info(pid, :binary)<br> total_size = bins |> Enum.reduce(0, fn {_id, size, _refs}, acc -> size + acc end)<br> {pid, total_size, length(bins)}<br> rescue<br> _ -> {pid, 0, 0}<br> end<br> end)<br> |> Enum.map(fn {pid, total_size, count} -><br> {pid, total_size, count}<br> end)<br> |> Enum.reduce(0, fn {_pid, total_size, _}, acc -> acc + total_size end)<br>end<br>```</div><div>The result returned by this function is in bytes 9600693, which is ~ 9 MB, and not even near the values of `binary` returned by `:erlang.memory()`.</div><div><br></div><div>Now, the weird part is that my app is running inside a docker container, and the docker container within a kubernetes cluster. In this setup I can see the problem, but if I'm running it locally, within bare bones docker, I can't see the same issue. And another thing I've noticed, even more weirder, is that when the BEAM starts, the binary memory is ~ 300 MB, and after each request it bumps up till it reaches ~ 900 MB, after which stays constant - what I mean is that the BEAM doesn't crash with out of memory, or something similar, but I'm still worried. Other applications I have, also running on Erlang 22, and exactly the same docker image, don't suffer from this problem, which makes me think this is related to the application code...</div><div>This is also happening on both Erlang 21, and 22.</div><div><br></div><div>Another thing to mention is that I used as well `recon`, and after running it, the binary section stays exactly the same, doesn't seem to be a matter of not being garbage collected.</div><div><br></div><div>My questions are:</div><div>1. What other type of binary (beside refc process binaries) can live in that zone of memory? Am I looking for the wrong thing?<br></div></div></blockquote><div><br></div><div>The main ones are binaries kept by ets tables and ports.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>2. How can I _look_ at the `binary` memory region to see what I have in there?<br></div></div></blockquote><div><br></div><div>You can use the instrument module in runtime_tools to get some more insights.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>3. Any other suggestion on what I might be doing wrong?</div></div></blockquote><div><br></div><div>From what you describe above I would look at the inet ports that are running and make sure that they are closed and that they use sane buffer sizes. </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>Any help will be much appreciated.</div><div><br></div><div>Thanks,</div><div>Alin</div></div>
</blockquote></div></div>