[erlang-questions] Process heap inspector

Björn-Egil Dahlberg <>
Mon Nov 28 15:47:29 CET 2011


On 2011-11-28 14:25, Michal Ptaszek wrote:
> So there is a chance to get it into official Erlang/OTP release? :)
I would say there is a chance ... a small one perhaps. =)

I agree with Kostis. It can be a very handy tool to inspect data in a 
process heap. At OTP (erts-team) we use gdb with plugins to inspect 
erlang process heaps when something goes horribly wrong. So it can be 
very useful.

I am not sure I would allow it in a production beam. If I had my way I 
would remove all tracing as well and only have it in a instrumented 
beam. I've seen production code use tracing to all sorts of interesting 
stuff. Bad practice (tm). It is too powerful.

(That said, I wouldn't really like to debug a live system without those 
tools)

I don't have a decisive opinion atm.

// Björn-Egil

>
> On Nov 28, 2011, at 2:05 PM, Björn-Egil Dahlberg wrote:
>
>> On 2011-11-28 12:11, Ahmed Omar wrote:
>>> I like the idea but i'm HORRIFIED about people misusing that and 
>>> actually try to read internal states within their programs :D
>>
>> I would put such a function in erts_debug:inspect_heap/1 instead.
>>
>> // Björn-Egil
>>
>>
>>>
>>> On Mon, Nov 28, 2011 at 12:02 PM, Michal Ptaszek 
>>> < 
>>> <mailto:>> wrote:
>>>
>>>     Hey,
>>>
>>>     good point, I knew I forgot about something - will add support
>>>     for that
>>>     really soon.
>>>
>>>     Kind regards,
>>>     Michal Ptaszek
>>>
>>>     On Nov 28, 2011, at 11:02 AM, Attila Rajmund Nohl wrote:
>>>
>>>     > Hello!
>>>     >
>>>     > I really like the idea. But shouldn't this list include the
>>>     message queue too?
>>>     >
>>>     > 2011/11/28 Michal Ptaszek <
>>>     <mailto:>>:
>>>     >> Hi everyone,
>>>     >>
>>>     >> This idea was born in my mind when debugging some complex,
>>>     live system
>>>     >> and trying to figure out where did all my memory go.
>>>     >>
>>>     >> So, when debugging live system/investigating suspicious
>>>     memory consumption patterns
>>>     >> or simply trying to understand better what's going on with
>>>     our processes, it might be useful
>>>     >> to take a peep at the data given process operates on.
>>>     >>
>>>     >> Right now it is possible to fetch internal gen_* processes
>>>     state via sys:get_status, we can do
>>>     >> some tracing (even using DTrace), we can also check
>>>     erlang:process_info output and analyze
>>>     >> it to become more or less familiar with what is the heap size
>>>     of our suspect. Still, not all processes
>>>     >> are OTP-compatible, and even if: we are going to get only
>>>     "alive" data coming from process' state
>>>     >> (not counting the outdated, not yet garbage collected terms).
>>>     Also, process_info informs us only
>>>     >> about allocated size of the heap, not about the actual usage
>>>     (although the pre-allocated chunks
>>>     >> are not available to the system, yet we might see how far we
>>>     are from growing/shrinking it).
>>>     >>
>>>     >> Enough with introduction, let's focus on the actual meat: my
>>>     idea was to create a new BIF,
>>>     >> namely erlang:inspect_heap(Pid) that allows us to take a look
>>>     at any process' heap, fetch the
>>>     >> terms residing there and check their actual size. So, for
>>>     instance:
>>>     >>
>>>     >>> ()12> S = erlang:inspect_heap(pid(0, 358, 0)).
>>>     >>> [{[[<<"5">>]|
>>>     >>>  
>>>      284735200226724471091958640173737944785062822211005333957298336375301959844499896296764925551414319236776784],
>>>     >>>   20},
>>>     >>>  {{'$internal_queue_len',0},3},
>>>     >>>  {{random_seed,{8236,26623,17360}},7},
>>>     >>>  {{'$ancestors',[ejabberd_c2s_sup,ejabberd_sup,<0.40.0>]},9},
>>>     >>>  {{'$initial_call',{gen,init_it,6}},7},
>>>     >>>  {{state,{socket_state,tls,
>>>     >>>                        {tlssock,#Port<0.3936>,#Port<0.3938>},
>>>     >>> <0.357.0>},
>>>     >>>        
>>>      ejabberd_socket,#Ref<0.0.0.10120>,false,<<"2855118401">>,
>>>     >>>          {sasl_state,"jabber",<<"pvp.net
>>>     <http://pvp.net/>">>,[],
>>>     >>>                    
>>>      #Fun<ejabberd_c2s.0.67315917>,#Fun<ejabberd_c2s.1.67315917>,
>>>     >>>                    
>>>      #Fun<ejabberd_c2s.2.67315917>,cyrsasl_digest,
>>>     >>>                      {state,5,<<"3598825873">>,
>>>     >>>                             {<<"dupa">>,<<...>>},
>>>     >>> <<>>,#Fun<ejabberd_c2s.0.67315917>,...}},
>>>     >>>          true,
>>>     >>>          {jid,<<"dupa">>,<<"pvp.net
>>>     <http://pvp.net/>">>,<<"hubbard">>,<<"dupa">>,
>>>     >>> <<"pvp.net <http://pvp.net/>">>,<<"hubbard">>},
>>>     >>> <<"Nicknamedupa">>,
>>>     >>>          {{1322,217197,749816},<0.358.0>},
>>>     >>>          {1,{<<"dupa">>,nil,nil}},
>>>     >>>          {1,{<<"dupa">>,nil,nil}},
>>>     >>>          {1,{<<"dupa">>,nil,nil}},
>>>     >>>          {xmlelement,<<"presence">>,[],
>>>     >>>                    
>>>      [{xmlcdata,<<...>>},{xmlelement,...},{...}|...]},
>>>     >>>          {userlist,none,[],false}},
>>>     >>>   564},
>>>     >>>  {{limits,undefined},3},
>>>     >>>  {{[],[]},3}]
>>>     >>
>>>     >> gives us a pretty good knowledge on <0.358.0>:
>>>     >> • '$_' - OTP + gen_fsm2 process dictionary stuff, 3, 9, 7
>>>     words each
>>>     >> • random_seed - obvious
>>>     >> • {{limits,undefined},3} - internal limits for gen_fsm2
>>>     message queue, 3 words
>>>     >> • {[], []} - most probably leftovers after fetching user's
>>>     presence lists
>>>     >> • {state, _} - gen_fsm2 state record - 564 words
>>>     >> • {[[<<"5">>]|, ...} - sequential tracing tokens? (I'm not
>>>     very familiar with that, I would
>>>     >> say that's something from our rootset
>>>     >>
>>>     >> The implementation is rather simple: if the process we probe
>>>     is not the caller one (we are not doing
>>>     >> erlang:inspect_heap(self()), the data is copied from the
>>>     callee heap to caller heap (to prevent from having
>>>     >> cross-process references in variables), then we compute flat
>>>     size of the each term we moved. Also, rootset
>>>     >> is also included in the summary (i.e. process dict, seq
>>>     tokens, etc.).
>>>     >>
>>>     >> Code is included in my inspect_heap OTP branch on:
>>>     >>  github: https://github.com/paulgray/otp/tree/inspect_heap
>>>     >>
>>>     >> I am still a little bit hesitant about suspending process we
>>>     probe: can anyone tell
>>>     >> me if acquiring main process lock would be enough to keep its
>>>     heap untouched during
>>>     >> the call?
>>>     >>
>>>     >> Please, do point any bugs and tell me what do you think about
>>>     the idea.
>>>     >>
>>>     >> Best regards,
>>>     >> Michal Ptaszek
>>>     >> _______________________________________________
>>>     >> erlang-questions mailing list
>>>     >>  <mailto:>
>>>     >> http://erlang.org/mailman/listinfo/erlang-questions
>>>     >>
>>>
>>>     _______________________________________________
>>>     erlang-questions mailing list
>>>      <mailto:>
>>>     http://erlang.org/mailman/listinfo/erlang-questions
>>>
>>>
>>>
>>>
>>> -- 
>>> Best Regards,
>>> - Ahmed Omar
>>> http://nl.linkedin.com/in/adiaa
>>> Follow me on twitter
>>> @spawn_think <http://twitter.com/#%21/spawn_think>
>>>
>>>
>>>
>>> _______________________________________________
>>> erlang-questions mailing list
>>> 
>>> http://erlang.org/mailman/listinfo/erlang-questions
>>
>> _______________________________________________
>> erlang-questions mailing list
>>  <mailto:>
>> http://erlang.org/mailman/listinfo/erlang-questions
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20111128/2f1b34c9/attachment.html>


More information about the erlang-questions mailing list