<html><head></head><body><div style="color:#000; background-color:#fff; font-family:Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:13px"><div id="yui_3_16_0_1_1490711189776_696454" dir="ltr"><span id="yui_3_16_0_1_1490711189776_696566">Just use the instrument module to track the actual terms held then reference the PID that created the term, set some hooks in beams allocator to track the pid references to the term (since instrument does not do this). Read beams memory to check what the actual term is.<br><br>All these statistics and reporting we currently have for erlang memory issues is simply poking at random things in the dark.</span></div> <div class="qtdSeparateBR"><br><br></div><div class="yahoo_quoted" style="display: block;"> <div style="font-family: Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif; font-size: 13px;"> <div style="font-family: HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif; font-size: 16px;"> <div dir="ltr"><font size="2" face="Arial"> On Thursday, April 6, 2017 10:41 AM, Jack Tang <himars@gmail.com> wrote:<br></font></div> <br><br> <div class="y_msg_container"><div id="yiv1167245695"><div><div dir="ltr">Hi Daniel<div><br clear="none"></div><div>Thank you for the detailed explanation and sharing the experiences. I focused on the performance of GC heavily before and ignored the allocation strategy. </div><div><br clear="none"></div><div>One more question, as you mentioned memory on eheap_alloc can be released by calling GC on every process. Right now I setup periodic gc process to free eheap. However can I set some process flag to make GC automatically?</div><div><br clear="none"></div><div>Thanks again!</div><div><br clear="none"></div><div>BR</div><div>-Jack</div><div><div class="yiv1167245695gmail_extra"><br clear="none"><div class="yiv1167245695yqt6880528993" id="yiv1167245695yqt90124"><div class="yiv1167245695gmail_quote">On Thu, Apr 6, 2017 at 7:52 PM, Dániel Szoboszlay <span dir="ltr"><<a rel="nofollow" shape="rect" ymailto="mailto:dszoboszlay@gmail.com" target="_blank" href="mailto:dszoboszlay@gmail.com">dszoboszlay@gmail.com</a>></span> wrote:<br clear="none"><blockquote class="yiv1167245695gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex;"><div dir="ltr">Hi Jack,<div><br clear="none"></div><div><b>tl;dr;</b> try starting Erlang with '<font face="monospace">+MEas<span class="yiv1167245695gmail-m_-642682847458610453inbox-inbox-Apple-converted-space"> </span>bf</font>'<br clear="none"><div><br clear="none"></div><div>Long explanation:</div><div><br clear="none"></div><div>If the memory is used by the <font face="monospace">ets_alloc</font> allocator, it can be either actively used by ETS tables (such as Mnesia <font face="monospace">disc_copies</font> tables) or it can be lost due to memory fragmentation. I guess this is memory fragmentation in your case, but let's quickly rule out the other possibility first. (By the way, in either case, garbage collecting processes won't reclaim the memory. GC operates on <font face="monospace">eheap_alloc</font> allocated memory.)</div><div><br clear="none"></div><div>So if <font face="monospace">erlang:memory(ets)</font> is close to what recon reports as allocated, the memory is in use. Your system wrote a lot of data to ETS/Mnesia, and didn't remove it afterwards. Inspect the contents of the tables and figure out what was left there that shouldn't be there, which part of the application should have removed that data and why didn't it do its job properly.</div><div><br clear="none"></div><div>The other, more likely option is fragmentation. I also experienced that the default memory allocation strategy (aoffcbf = address order first fit carrier best fit) can perform poorly when you use a lot of memory. All address order first fit strategies will use heavily the multiblock carriers with the lowest memory addresses, and if you have many carriers, those placed higher in memory will have less and less chance to be used. In my particular case <font face="monospace">ets_alloc</font> created a total of 150k multiblock carriers for storing ~1.2TB data in ETS tables. This resulted in ~100 GB unused memory being wasted in the high address carriers. You have a much smaller system, but depending on the usage patterns of your ETS data you can end up in a similar situation.</div><div><br clear="none"></div><div>You can check the number of carriers with <font face="monospace">erlang:system_info({allocator, ets_alloc})</font>. It will print something like this:</div><div><div><font face="monospace">[{instance,0,</font></div><div><font face="monospace"> [{versions,"0.9","3.0"},</font></div><div><font face="monospace"> {options,[...]},</font></div><div><font face="monospace"> {mbcs,[...</font></div><div><font face="monospace"> {carriers,<font color="#ff0000">1</font>,1,1}, %% <font color="#ff0000"><- number of multi block carriers = 1</font></font></div><div><font face="monospace"> ...]},</font></div><div><font face="monospace"> {sbcs,[...</font></div><div><font face="monospace"> {carriers,<font color="#ff0000">0</font>,0,0},<span class="yiv1167245695gmail-m_-642682847458610453inbox-inbox-Apple-converted-space"> </span> %% <font color="#ff0000"><- number of single block carriers = 0</font></font></div><div><font face="monospace"> ...]},</font></div><div><font face="monospace"> {calls,[...]}]},</font></div><div><font face="monospace"> {instance,1,...</font></div></div><div>Check theaw numbera across all your allocator instances. You will typically have very few single block carriers (unless you store huge records in ETS). In my experience, fragmentation correlates well with the number of carriers an allocator handles, and can become quite significant above ~10 carriers.</div><div><br clear="none"></div><div>So, If you have the same problem I described, I have some bad news and some good news. The bad news is that I don't know a way of forcing the VM to defragment memory and get rid of the waste. The good news is that the bf (best fit) allocation strategy (which used to be the default up to R16) performs much better when you have many carriers. You need to pass the '<font face="monospace">+MEas bf</font>' command line argument to the VM to switch <font face="monospace">ets_alloc</font> to bf strategy.</div><div><br clear="none"></div><div>Hope it helps,</div><div>Daniel</div></div></div><div class="yiv1167245695gmail-HOEnZb"><div class="yiv1167245695gmail-h5"><br clear="none"><div class="yiv1167245695gmail_quote"><div dir="ltr">On Sat, 1 Apr 2017 at 05:36 Jack Tang <<a rel="nofollow" shape="rect" ymailto="mailto:himars@gmail.com" target="_blank" href="mailto:himars@gmail.com">himars@gmail.com</a>> wrote:<br clear="none"></div><blockquote class="yiv1167245695gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex;"><div class="yiv1167245695gmail-m_-642682847458610453gmail_msg" dir="ltr"><span class="yiv1167245695gmail-m_-642682847458610453gmail_msg" style="font-size:12.8px;">After setting up erlang memory visualization, we find etc allocator does not release the memory during some historical datum are remove from mnesia tables. Can I release the memory on the fly rather than restart the mnesia application? Thanks!</span><br clear="none" class="yiv1167245695gmail-m_-642682847458610453gmail_msg"><div class="yiv1167245695gmail-m_-642682847458610453gmail_msg"><span class="yiv1167245695gmail-m_-642682847458610453gmail_msg" style="font-size:12.8px;"><img class="yiv1167245695gmail-m_-642682847458610453gmail_msg" src="https://pbs.twimg.com/media/C8TBNK7VwAIMme7.jpg" width="536" height="308"><br clear="none" class="yiv1167245695gmail-m_-642682847458610453gmail_msg"></span></div><div class="yiv1167245695gmail-m_-642682847458610453gmail_msg"></div><div class="yiv1167245695gmail_extra yiv1167245695gmail-m_-642682847458610453gmail_msg"><span class="yiv1167245695gmail-m_-642682847458610453gmail_msg" style="font-size:12.8px;"><img class="yiv1167245695gmail-m_-642682847458610453gmail_msg" src="https://pbs.twimg.com/media/C8TBNK8UwAEhvGL.jpg" width="536" height="178"><br clear="none" class="yiv1167245695gmail-m_-642682847458610453gmail_msg"><br clear="none" class="yiv1167245695gmail-m_-642682847458610453gmail_msg"></span></div><div class="yiv1167245695gmail_extra yiv1167245695gmail-m_-642682847458610453gmail_msg"><div class="yiv1167245695gmail_quote yiv1167245695gmail-m_-642682847458610453gmail_msg"><br clear="none" class="yiv1167245695gmail-m_-642682847458610453gmail_msg"></div><div class="yiv1167245695gmail_quote yiv1167245695gmail-m_-642682847458610453gmail_msg">BR</div><div class="yiv1167245695gmail_quote yiv1167245695gmail-m_-642682847458610453gmail_msg"><br clear="none" class="yiv1167245695gmail-m_-642682847458610453gmail_msg"></div><div class="yiv1167245695gmail_quote yiv1167245695gmail-m_-642682847458610453gmail_msg"></div></div></div><div class="yiv1167245695gmail-m_-642682847458610453gmail_msg" dir="ltr"><div class="yiv1167245695gmail_extra yiv1167245695gmail-m_-642682847458610453gmail_msg"><div class="yiv1167245695gmail_quote yiv1167245695gmail-m_-642682847458610453gmail_msg">On Sun, Jan 15, 2017 at 4:47 AM, Dániel Szoboszlay <span class="yiv1167245695gmail-m_-642682847458610453gmail_msg" dir="ltr"><<a rel="nofollow" shape="rect" class="yiv1167245695gmail-m_-642682847458610453gmail_msg" ymailto="mailto:dszoboszlay@gmail.com" target="_blank" href="mailto:dszoboszlay@gmail.com">dszoboszlay@gmail.com</a>></span> wrote:<br clear="none" class="yiv1167245695gmail-m_-642682847458610453gmail_msg"></div></div></div><div class="yiv1167245695gmail-m_-642682847458610453gmail_msg" dir="ltr"><div class="yiv1167245695gmail_extra yiv1167245695gmail-m_-642682847458610453gmail_msg"><div class="yiv1167245695gmail_quote yiv1167245695gmail-m_-642682847458610453gmail_msg"><blockquote class="yiv1167245695gmail_quote yiv1167245695gmail-m_-642682847458610453gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex;"><div class="yiv1167245695gmail-m_-642682847458610453gmail_msg" dir="ltr">Hi Jack,<div class="yiv1167245695gmail-m_-642682847458610453gmail_msg"><br clear="none" class="yiv1167245695gmail-m_-642682847458610453gmail_msg"></div><div class="yiv1167245695gmail-m_-642682847458610453gmail_msg">I guess the 9 GB is lost due to memory fragmentation. Erlang allocates memory in large chunks called carriers from the OS, then places the blocks your program actually needs on these carriers. A carrier can only be returned to the OS once all the blocks on it have been freed (and even then, the memory allocator may decide to keep it around for a while in case more memory is needed).</div><div class="yiv1167245695gmail-m_-642682847458610453gmail_msg"><br clear="none" class="yiv1167245695gmail-m_-642682847458610453gmail_msg"></div><div class="yiv1167245695gmail-m_-642682847458610453gmail_msg">You can check with <a rel="nofollow" shape="rect" class="yiv1167245695gmail-m_-642682847458610453gmail_msg" target="_blank" href="https://ferd.github.io/recon/recon_alloc.html">recon_alloc</a> how much unused memory is lost due to fragmentation in the various allocators.</div><div class="yiv1167245695gmail-m_-642682847458610453gmail_msg"><br clear="none" class="yiv1167245695gmail-m_-642682847458610453gmail_msg"></div><div class="yiv1167245695gmail-m_-642682847458610453gmail_msg">The bad news is that you cannot defragment the carriers, and if the selected memory allocator strategy doesn't work well for your application, you cannot change it either without restarting the emulator.</div><div class="yiv1167245695gmail-m_-642682847458610453gmail_msg"><br clear="none" class="yiv1167245695gmail-m_-642682847458610453gmail_msg"></div><div class="yiv1167245695gmail-m_-642682847458610453gmail_msg">However, if the memory is wasted in the <font class="yiv1167245695gmail-m_-642682847458610453gmail_msg" face="monospace">eheap_alloc</font>, you may try to force a GC on all processes a couple of times. As the GC copies the memory, it will allocate new blocks and free up the old heap blocks. So there's a chance the allocators can compact the blocks together on fewer segments. But that's just a guess, it may or may not work at all.</div><div class="yiv1167245695gmail-m_-642682847458610453gmail_msg"><br clear="none" class="yiv1167245695gmail-m_-642682847458610453gmail_msg"></div><div class="yiv1167245695gmail-m_-642682847458610453gmail_msg">Cheers,</div><div class="yiv1167245695gmail-m_-642682847458610453gmail_msg">Daniel</div></div><br clear="none" class="yiv1167245695gmail-m_-642682847458610453gmail_msg"><div class="yiv1167245695gmail_quote yiv1167245695gmail-m_-642682847458610453gmail_msg"><div class="yiv1167245695gmail-m_-642682847458610453gmail_msg"><div class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361h5 yiv1167245695gmail-m_-642682847458610453gmail_msg"><div class="yiv1167245695gmail-m_-642682847458610453gmail_msg" dir="ltr">On Sat, 14 Jan 2017 at 08:04 Jack Tang <<a rel="nofollow" shape="rect" class="yiv1167245695gmail-m_-642682847458610453gmail_msg" ymailto="mailto:himars@gmail.com" target="_blank" href="mailto:himars@gmail.com">himars@gmail.com</a>> wrote:<br clear="none" class="yiv1167245695gmail-m_-642682847458610453gmail_msg"></div></div></div><blockquote class="yiv1167245695gmail_quote yiv1167245695gmail-m_-642682847458610453gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex;"><div class="yiv1167245695gmail-m_-642682847458610453gmail_msg"><div class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361h5 yiv1167245695gmail-m_-642682847458610453gmail_msg"><div class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg yiv1167245695gmail-m_-642682847458610453gmail_msg" dir="ltr"><span class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg yiv1167245695gmail-m_-642682847458610453gmail_msg" style="font-size:12.8px;">Hello list,</span><div class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg yiv1167245695gmail-m_-642682847458610453gmail_msg" style="font-size:12.8px;"><br clear="none" class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg yiv1167245695gmail-m_-642682847458610453gmail_msg"></div><div class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg yiv1167245695gmail-m_-642682847458610453gmail_msg" style="font-size:12.8px;">I run one Erlang application on Debian server and today I find the beam process consumes around 35G memory by `top` command.</div><div class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg yiv1167245695gmail-m_-642682847458610453gmail_msg" style="font-size:12.8px;"><br clear="none" class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg yiv1167245695gmail-m_-642682847458610453gmail_msg"></div><div class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg yiv1167245695gmail-m_-642682847458610453gmail_msg" style="font-size:12.8px;"><div class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg yiv1167245695gmail-m_-642682847458610453gmail_msg">```</div><div class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg yiv1167245695gmail-m_-642682847458610453gmail_msg">KiB Mem: 99194912 total, 61682656 used, 37512252 free, 397380 buffers</div><div class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg yiv1167245695gmail-m_-642682847458610453gmail_msg">KiB Swap: 0 total, 0 used, 0 free. 18684864 cached Mem</div><div class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg yiv1167245695gmail-m_-642682847458610453gmail_msg"><br clear="none" class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg yiv1167245695gmail-m_-642682847458610453gmail_msg"></div><div class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg yiv1167245695gmail-m_-642682847458610453gmail_msg"> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND</div><div class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg yiv1167245695gmail-m_-642682847458610453gmail_msg">11858 usr1 20 0 36.850g 0.032t 6220 S 73.5 34.4 8038:49 beam.smp</div><div class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg yiv1167245695gmail-m_-642682847458610453gmail_msg">```</div><div class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg yiv1167245695gmail-m_-642682847458610453gmail_msg"><br clear="none" class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg yiv1167245695gmail-m_-642682847458610453gmail_msg"></div><div class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg yiv1167245695gmail-m_-642682847458610453gmail_msg">I connect to the Erlang application using remote shell and find the mem-leaked supervisor tree and run gc on the whole tree. Code looks like blow:</div><div class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg yiv1167245695gmail-m_-642682847458610453gmail_msg"><br clear="none" class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg yiv1167245695gmail-m_-642682847458610453gmail_msg"></div><div class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg yiv1167245695gmail-m_-642682847458610453gmail_msg">```</div><div class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg yiv1167245695gmail-m_-642682847458610453gmail_msg">lists:foreach(fun(E) -> PId = element(2, E), erlang:garbage_collect(PId) end, supervisor:which_children( some_thing_sup)).<br clear="none" class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg yiv1167245695gmail-m_-642682847458610453gmail_msg"></div><div class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg yiv1167245695gmail-m_-642682847458610453gmail_msg">```</div><div class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg yiv1167245695gmail-m_-642682847458610453gmail_msg"><br clear="none" class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg yiv1167245695gmail-m_-642682847458610453gmail_msg"></div><div class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg yiv1167245695gmail-m_-642682847458610453gmail_msg">and erlang:memory() decreases from 32G to 23G.</div><div class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg yiv1167245695gmail-m_-642682847458610453gmail_msg">```</div><div class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg yiv1167245695gmail-m_-642682847458610453gmail_msg"><div class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg yiv1167245695gmail-m_-642682847458610453gmail_msg">[{total,22982011544},</div><div class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg yiv1167245695gmail-m_-642682847458610453gmail_msg"> {processes,12884182336},</div><div class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg yiv1167245695gmail-m_-642682847458610453gmail_msg"> {processes_used,12884170336},</div><div class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg yiv1167245695gmail-m_-642682847458610453gmail_msg"> {system,10097829208},</div><div class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg yiv1167245695gmail-m_-642682847458610453gmail_msg"> {atom,13828705},</div><div class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg yiv1167245695gmail-m_-642682847458610453gmail_msg"> {atom_used,13796692},</div><div class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg yiv1167245695gmail-m_-642682847458610453gmail_msg"> {binary,170530288},</div><div class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg yiv1167245695gmail-m_-642682847458610453gmail_msg"> {code,16450626},</div><div class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg yiv1167245695gmail-m_-642682847458610453gmail_msg"> {ets,9637717576}]</div></div><div class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg yiv1167245695gmail-m_-642682847458610453gmail_msg">```</div><div class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg yiv1167245695gmail-m_-642682847458610453gmail_msg"><br clear="none" class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg yiv1167245695gmail-m_-642682847458610453gmail_msg"></div><div class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg yiv1167245695gmail-m_-642682847458610453gmail_msg">However, when I input `top` command, the beam process still takes 35G memory. What can I do to release the 9G memory? Thanks</div><div class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg yiv1167245695gmail-m_-642682847458610453gmail_msg"><br clear="none" class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg yiv1167245695gmail-m_-642682847458610453gmail_msg"></div><div class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg yiv1167245695gmail-m_-642682847458610453gmail_msg">BR</div><div class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg yiv1167245695gmail-m_-642682847458610453gmail_msg">-Jack</div></div></div></div></div>
______________________________ _________________<br clear="none" class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg yiv1167245695gmail-m_-642682847458610453gmail_msg">
erlang-questions mailing list<br clear="none" class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg yiv1167245695gmail-m_-642682847458610453gmail_msg">
<a rel="nofollow" shape="rect" class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg yiv1167245695gmail-m_-642682847458610453gmail_msg" ymailto="mailto:erlang-questions@erlang.org" target="_blank" href="mailto:erlang-questions@erlang.org">erlang-questions@erlang.org</a><br clear="none" class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg yiv1167245695gmail-m_-642682847458610453gmail_msg">
<a rel="nofollow" shape="rect" class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg yiv1167245695gmail-m_-642682847458610453gmail_msg" target="_blank" href="http://erlang.org/mailman/listinfo/erlang-questions">http://erlang.org/mailman/ listinfo/erlang-questions</a><br clear="none" class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg yiv1167245695gmail-m_-642682847458610453gmail_msg">
</blockquote></div>
</blockquote></div></div></div><div class="yiv1167245695gmail-m_-642682847458610453gmail_msg" dir="ltr"><div class="yiv1167245695gmail_extra yiv1167245695gmail-m_-642682847458610453gmail_msg"><div class="yiv1167245695gmail-m_-642682847458610453gmail_msg"><br clear="none" class="yiv1167245695gmail-m_-642682847458610453gmail_msg"></div>-- <br clear="none" class="yiv1167245695gmail-m_-642682847458610453gmail_msg"><div class="yiv1167245695gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361gmail_signature yiv1167245695gmail-m_-642682847458610453gmail_msg">Jack Tang<br clear="none" class="yiv1167245695gmail-m_-642682847458610453gmail_msg"><br clear="none" class="yiv1167245695gmail-m_-642682847458610453gmail_msg"><br clear="none" class="yiv1167245695gmail-m_-642682847458610453gmail_msg"><a rel="nofollow" shape="rect" class="yiv1167245695gmail-m_-642682847458610453gmail_msg" target="_blank" href="http://www.linkedin.com/in/jacktang">http://www.linkedin.com/in/ jacktang</a><br clear="none" class="yiv1167245695gmail-m_-642682847458610453gmail_msg"></div>
</div></div>
______________________________ _________________<br clear="none" class="yiv1167245695gmail-m_-642682847458610453gmail_msg">
erlang-questions mailing list<br clear="none" class="yiv1167245695gmail-m_-642682847458610453gmail_msg">
<a rel="nofollow" shape="rect" class="yiv1167245695gmail-m_-642682847458610453gmail_msg" ymailto="mailto:erlang-questions@erlang.org" target="_blank" href="mailto:erlang-questions@erlang.org">erlang-questions@erlang.org</a><br clear="none" class="yiv1167245695gmail-m_-642682847458610453gmail_msg">
<a rel="nofollow" shape="rect" class="yiv1167245695gmail-m_-642682847458610453gmail_msg" target="_blank" href="http://erlang.org/mailman/listinfo/erlang-questions">http://erlang.org/mailman/ listinfo/erlang-questions</a><br clear="none" class="yiv1167245695gmail-m_-642682847458610453gmail_msg">
</blockquote></div>
</div></div></blockquote></div></div><br clear="none"><br clear="all"><div><br clear="none"></div>-- <br clear="none"><div class="yiv1167245695gmail_signature">Jack Tang<br clear="none"><br clear="none"><br clear="none"><a rel="nofollow" shape="rect" target="_blank" href="http://www.linkedin.com/in/jacktang">http://www.linkedin.com/in/jacktang</a><br clear="none"></div>
</div></div></div></div></div><div class="yqt6880528993" id="yqt34043">_______________________________________________<br clear="none">erlang-questions mailing list<br clear="none"><a shape="rect" ymailto="mailto:erlang-questions@erlang.org" href="mailto:erlang-questions@erlang.org">erlang-questions@erlang.org</a><br clear="none"><a shape="rect" href="http://erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br clear="none"></div><br><br></div> </div> </div> </div></div></body></html>