[erlang-questions] Where are the binaries in message queue but not part of process_info(Pid, binary) ?

[Bekes, Andras G]

Hi All,

While investigating a system that transfers lots of refc binaries, I tried measuring the amount of data in the message queue of a process. I noticed that the binaries in the message queue are not always included in the result of process_info(Pid, binary).
In particular, when observing the binaries, I see time intervals when each of the new messages appear in the binary list, and time intervals when the binary list stays the same regardless of new messages in the queue.

I made a simple module that demonstrates the phenomenon. The output looks like:

Msgs sent: 0 Msgs in queue: 0 Binaries of receiver: 0
Msgs sent: 1 Msgs in queue: 1 Binaries of receiver: 1
Msgs sent: 2 Msgs in queue: 2 Binaries of receiver: 2
[...goes well up to 36...]
Msgs sent: 36 Msgs in queue: 36 Binaries of receiver: 36
[then binary count stays 36, for thousands of messages, and suddenly starts counting again ! ]

Looks like if there are some magic limits at 36 (sometimes 33), 256, 4781, 12518 and 20255 messages/binaries and so far I was unable to find any logic in which limit takes effect.

I am aware that this feature of process_info is not meant for production use ("can be changed or removed without prior notice."), but this misleading behavior is maybe worse than not having it.

May I kindly ask for an explanation of what I observe?

Thanks!

Andras G. Bekes

----------------------------------------------------------------------------------
My test module:

-module(msgq_len_bin_test).

-export([test_sleeper/0,test_receiver/0]).

test_sleeper()->
   io:format("Testing sleeper...\n",[]),
   PID = spawn_link(fun sleeper/0),
   sender(PID,0,10000,x). % no reason to test further, binary count never goes above 36

test_receiver()->
   io:format("Testing receiver...\n",[]),
   PID = spawn_link(fun receiver/0),
   sender(PID,0,100000,x). % worth testing big numbers

sleeper()->timer:sleep(100000000).

receiver()->
   timer:sleep(10),
   receive _ -> ok end,
   receiver().

sender(_,N,N,_)-> io:format("Sent ~p messages.",[N]);
sender(PID,N,Max,LastBinLen)->

   [{message_queue_len,MsgQLen}, {binary,BinaryInfoList}] =
      erlang:process_info(PID,[message_queue_len,binary]),

   BinLen = length(BinaryInfoList),
   case BinLen of
      LastBinLen -> io:format(".",[]);
      _ -> io:format("Msgs sent: ~p Msgs in queue: ~p Binaries of receiver: ~p \n",[N,MsgQLen,BinLen])
   end,

   RefcBinary = binary:copy(<<N:32>>,1000),
   PID ! RefcBinary,

   sender(PID,N+1,Max,BinLen).



________________________________

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20171017/1000f971/attachment.htm>