[erlang-questions] Where are the binaries in message queue but not part of process_info(Pid, binary) ?

[Bekes, Andras G]

Hi Lukas,

Thank you very much for the quick and detailed reply. This explains it.

Now I have a few further questions.

- It seems the reliable way to measure the amount of data in the message queue is setting message_queue_data process flag to off_heap beforehand (ideally at spawn time with spawn_opt), and retrieve the full message queue content with process_info(Pid,messages) – which must be fully scanned. Is there a better solution? (I originally aimed at refc binaries only because I just wanted a quick estimation, and in my case anything else was negligible.)

- May I suggest to improve the documentation of erlang:process_info(Pid, binary) by the below facts, also referring to message_queue_data process flag?

- Is it possible to implement a new process_info_item() of erlang:process_info/2 that gives information about binaries currently in the message queue but off-heap?

- The documentation suggests using off_heap message_queue_data “if the process potentially can get many messages”, but this is rather vague. Does this refer to processes expected to have many *queued* messages? I guess a process receiving but not queuing many messages performs better with on_heap (“Performance of the actual message passing is however generally better when not using flag off_heap.”).

Thanks,
   Andras

From: Lukas Larsson [mailto:lukas@REDACTED]
Sent: Tuesday, October 17, 2017 4:37 PM
To: Bekes, Andras G (IST)
Cc: erlang-questions@REDACTED
Subject: Re: [erlang-questions] Where are the binaries in message queue but not part of process_info(Pid, binary) ?

Hello,

On Tue, Oct 17, 2017 at 4:00 PM, Bekes, Andras G <Andras.Bekes@REDACTED<mailto:Andras.Bekes@REDACTED>> wrote:
While investigating a system that transfers lots of refc binaries, I tried measuring the amount of data in the message queue of a process. I noticed that the binaries in the message queue are not always included in the result of process_info(Pid, binary).
In particular, when observing the binaries, I see time intervals when each of the new messages appear in the binary list, and time intervals when the binary list stays the same regardless of new messages in the queue.

erlang:process_info(Pid, binary) returns the binaries that are currently on the heap of the process. Binaries in messages that have not yet been received may or may not be part of the heap depending on internal emulator optimizations.

The message_queue_data process flag explains this a little more and also gives a way to control it to some extent.

http://erlang.org/doc/man/erlang.html#process_flag_message_queue_data

Lukas



________________________________


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20171017/c0feafae/attachment.htm>