<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">Hi Jack,<br>
<br>
The simplest way to make an Erlang process automatically GC is to
use a temporary Erlang process, since the death of the Erlang
process will cause GC to occur soon after. It is common to avoid
long-lived Erlang processes that create lots of temporary data
quickly due to excessive memory consumption, and to refactor the
long-lived process a temporary process is used to create the
temporary data within (this matters at high request rates, though
you may not see the problem with low request rates). You may also
tune an Erlang process with fullsweep_after to cause GC to occur
more frequently, but I see that as only performance tuning, not
fixing a problem that should be fixed with a temporary Erlang
process. There also is the erlang:garbage_collect function,
though you shouldn't use that in production source code (source
code that needs to be dependable as its expectations change).<br>
<br>
Best Regards,<br>
Michael<br>
<br>
On 04/06/2017 07:41 AM, Jack Tang wrote:<br>
</div>
<blockquote
cite="mid:CAC+_AnXGvspnBOCRSju7L9V_E56WgLpJggoOnNX95UG91qEfqQ@mail.gmail.com"
type="cite">
<div dir="ltr">Hi Daniel
<div><br>
</div>
<div>Thank you for the detailed explanation and sharing the
experiences. I focused on the performance of GC heavily before
and ignored the allocation strategy. </div>
<div><br>
</div>
<div>One more question, as you mentioned memory on eheap_alloc
can be released by calling GC on every process. Right now I
setup periodic gc process to free eheap. However can I set
some process flag to make GC automatically?</div>
<div><br>
</div>
<div>Thanks again!</div>
<div><br>
</div>
<div>BR</div>
<div>-Jack</div>
<div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Thu, Apr 6, 2017 at 7:52 PM,
Dániel Szoboszlay <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:dszoboszlay@gmail.com" target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:dszoboszlay@gmail.com">dszoboszlay@gmail.com</a></a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Hi Jack,
<div><br>
</div>
<div><b>tl;dr;</b> try starting Erlang with '<font
face="monospace">+MEas<span
class="gmail-m_-642682847458610453inbox-inbox-Apple-converted-space"> </span>bf</font>'<br>
<div><br>
</div>
<div>Long explanation:</div>
<div><br>
</div>
<div>If the memory is used by the <font
face="monospace">ets_alloc</font> allocator, it
can be either actively used by ETS tables (such as
Mnesia <font face="monospace">disc_copies</font>
tables) or it can be lost due to memory
fragmentation. I guess this is memory
fragmentation in your case, but let's quickly rule
out the other possibility first. (By the way, in
either case, garbage collecting processes won't
reclaim the memory. GC operates on <font
face="monospace">eheap_alloc</font> allocated
memory.)</div>
<div><br>
</div>
<div>So if <font face="monospace">erlang:memory(ets)</font>
is close to what recon reports as allocated, the
memory is in use. Your system wrote a lot of data
to ETS/Mnesia, and didn't remove it afterwards.
Inspect the contents of the tables and figure out
what was left there that shouldn't be there, which
part of the application should have removed that
data and why didn't it do its job properly.</div>
<div><br>
</div>
<div>The other, more likely option is fragmentation.
I also experienced that the default memory
allocation strategy (aoffcbf = address order first
fit carrier best fit) can perform poorly when you
use a lot of memory. All address order first fit
strategies will use heavily the multiblock
carriers with the lowest memory addresses, and if
you have many carriers, those placed higher in
memory will have less and less chance to be used.
In my particular case <font face="monospace">ets_alloc</font>
created a total of 150k multiblock carriers for
storing ~1.2TB data in ETS tables. This resulted
in ~100 GB unused memory being wasted in the high
address carriers. You have a much smaller system,
but depending on the usage patterns of your ETS
data you can end up in a similar situation.</div>
<div><br>
</div>
<div>You can check the number of carriers with <font
face="monospace">erlang:system_info({allocator,
ets_alloc})</font>. It will print something like
this:</div>
<div>
<div><font face="monospace">[{instance,0,</font></div>
<div><font face="monospace">
[{versions,"0.9","3.0"},</font></div>
<div><font face="monospace">
{options,[...]},</font></div>
<div><font face="monospace"> {mbcs,[...</font></div>
<div><font face="monospace">
{carriers,<font color="#ff0000">1</font>,1,1},
%% <font color="#ff0000"><- number of
multi block carriers = 1</font></font></div>
<div><font face="monospace">
...]},</font></div>
<div><font face="monospace"> {sbcs,[...</font></div>
<div><font face="monospace">
{carriers,<font color="#ff0000">0</font>,0,0},<span
class="gmail-m_-642682847458610453inbox-inbox-Apple-converted-space"> </span> %%
<font color="#ff0000"><- number of single
block carriers = 0</font></font></div>
<div><font face="monospace">
...]},</font></div>
<div><font face="monospace">
{calls,[...]}]},</font></div>
<div><font face="monospace"> {instance,1,...</font></div>
</div>
<div>Check theaw numbera across all your allocator
instances. You will typically have very few single
block carriers (unless you store huge records in
ETS). In my experience, fragmentation correlates
well with the number of carriers an allocator
handles, and can become quite significant above
~10 carriers.</div>
<div><br>
</div>
<div>So, If you have the same problem I described, I
have some bad news and some good news. The bad
news is that I don't know a way of forcing the VM
to defragment memory and get rid of the waste. The
good news is that the bf (best fit) allocation
strategy (which used to be the default up to R16)
performs much better when you have many carriers.
You need to pass the '<font face="monospace">+MEas
bf</font>' command line argument to the VM to
switch <font face="monospace">ets_alloc</font> to
bf strategy.</div>
<div><br>
</div>
<div>Hope it helps,</div>
<div>Daniel</div>
</div>
</div>
<div class="gmail-HOEnZb">
<div class="gmail-h5"><br>
<div class="gmail_quote">
<div dir="ltr">On Sat, 1 Apr 2017 at 05:36 Jack
Tang <<a moz-do-not-send="true"
href="mailto:himars@gmail.com" target="_blank">himars@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px
0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr"
class="gmail-m_-642682847458610453gmail_msg"><span
style="font-size:12.8px"
class="gmail-m_-642682847458610453gmail_msg">After
setting up erlang memory visualization, we
find etc allocator does not release the
memory during some historical datum are
remove from mnesia tables. Can I release the
memory on the fly rather than restart the
mnesia application? Thanks!</span><br
class="gmail-m_-642682847458610453gmail_msg">
<div
class="gmail-m_-642682847458610453gmail_msg"><span
style="font-size:12.8px"
class="gmail-m_-642682847458610453gmail_msg"><img
moz-do-not-send="true"
src="https://pbs.twimg.com/media/C8TBNK7VwAIMme7.jpg"
class="gmail-m_-642682847458610453gmail_msg" height="308" width="536"><br
class="gmail-m_-642682847458610453gmail_msg">
</span></div>
<div class="gmail_extra
gmail-m_-642682847458610453gmail_msg"><span
style="font-size:12.8px"
class="gmail-m_-642682847458610453gmail_msg"><img
moz-do-not-send="true"
src="https://pbs.twimg.com/media/C8TBNK8UwAEhvGL.jpg"
class="gmail-m_-642682847458610453gmail_msg" height="178" width="536"><br
class="gmail-m_-642682847458610453gmail_msg">
<br
class="gmail-m_-642682847458610453gmail_msg">
</span></div>
<div class="gmail_extra
gmail-m_-642682847458610453gmail_msg">
<div class="gmail_quote
gmail-m_-642682847458610453gmail_msg"><br
class="gmail-m_-642682847458610453gmail_msg">
</div>
<div class="gmail_quote
gmail-m_-642682847458610453gmail_msg">BR</div>
<div class="gmail_quote
gmail-m_-642682847458610453gmail_msg"><br
class="gmail-m_-642682847458610453gmail_msg">
</div>
</div>
</div>
<div dir="ltr"
class="gmail-m_-642682847458610453gmail_msg">
<div class="gmail_extra
gmail-m_-642682847458610453gmail_msg">
<div class="gmail_quote
gmail-m_-642682847458610453gmail_msg">On
Sun, Jan 15, 2017 at 4:47 AM, Dániel
Szoboszlay <span dir="ltr"
class="gmail-m_-642682847458610453gmail_msg"><<a
moz-do-not-send="true"
href="mailto:dszoboszlay@gmail.com"
class="gmail-m_-642682847458610453gmail_msg"
target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:dszoboszlay@gmail.com">dszoboszlay@gmail.com</a></a>></span>
wrote:<br
class="gmail-m_-642682847458610453gmail_msg">
</div>
</div>
</div>
<div dir="ltr"
class="gmail-m_-642682847458610453gmail_msg">
<div class="gmail_extra
gmail-m_-642682847458610453gmail_msg">
<div class="gmail_quote
gmail-m_-642682847458610453gmail_msg">
<blockquote class="gmail_quote
gmail-m_-642682847458610453gmail_msg"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr"
class="gmail-m_-642682847458610453gmail_msg">Hi
Jack,
<div
class="gmail-m_-642682847458610453gmail_msg"><br
class="gmail-m_-642682847458610453gmail_msg">
</div>
<div
class="gmail-m_-642682847458610453gmail_msg">I
guess the 9 GB is lost due to memory
fragmentation. Erlang allocates
memory in large chunks called
carriers from the OS, then places
the blocks your program actually
needs on these carriers. A carrier
can only be returned to the OS once
all the blocks on it have been freed
(and even then, the memory allocator
may decide to keep it around for a
while in case more memory is
needed).</div>
<div
class="gmail-m_-642682847458610453gmail_msg"><br
class="gmail-m_-642682847458610453gmail_msg">
</div>
<div
class="gmail-m_-642682847458610453gmail_msg">You
can check with <a
moz-do-not-send="true"
href="https://ferd.github.io/recon/recon_alloc.html"
class="gmail-m_-642682847458610453gmail_msg" target="_blank">recon_alloc</a> how
much unused memory is lost due to
fragmentation in the various
allocators.</div>
<div
class="gmail-m_-642682847458610453gmail_msg"><br
class="gmail-m_-642682847458610453gmail_msg">
</div>
<div
class="gmail-m_-642682847458610453gmail_msg">The
bad news is that you cannot
defragment the carriers, and if the
selected memory allocator strategy
doesn't work well for your
application, you cannot change it
either without restarting the
emulator.</div>
<div
class="gmail-m_-642682847458610453gmail_msg"><br
class="gmail-m_-642682847458610453gmail_msg">
</div>
<div
class="gmail-m_-642682847458610453gmail_msg">However,
if the memory is wasted in the <font
class="gmail-m_-642682847458610453gmail_msg" face="monospace">eheap_alloc</font>,
you may try to force a GC on all
processes a couple of times. As the
GC copies the memory, it will
allocate new blocks and free up the
old heap blocks. So there's a chance
the allocators can compact the
blocks together on fewer segments.
But that's just a guess, it may or
may not work at all.</div>
<div
class="gmail-m_-642682847458610453gmail_msg"><br
class="gmail-m_-642682847458610453gmail_msg">
</div>
<div
class="gmail-m_-642682847458610453gmail_msg">Cheers,</div>
<div
class="gmail-m_-642682847458610453gmail_msg">Daniel</div>
</div>
<br
class="gmail-m_-642682847458610453gmail_msg">
<div class="gmail_quote
gmail-m_-642682847458610453gmail_msg">
<div
class="gmail-m_-642682847458610453gmail_msg">
<div
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361h5
gmail-m_-642682847458610453gmail_msg">
<div dir="ltr"
class="gmail-m_-642682847458610453gmail_msg">On
Sat, 14 Jan 2017 at 08:04 Jack
Tang <<a
moz-do-not-send="true"
href="mailto:himars@gmail.com"
class="gmail-m_-642682847458610453gmail_msg" target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:himars@gmail.com">himars@gmail.com</a></a>>
wrote:<br
class="gmail-m_-642682847458610453gmail_msg">
</div>
</div>
</div>
<blockquote class="gmail_quote
gmail-m_-642682847458610453gmail_msg"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div
class="gmail-m_-642682847458610453gmail_msg">
<div
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361h5
gmail-m_-642682847458610453gmail_msg">
<div dir="ltr"
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg
gmail-m_-642682847458610453gmail_msg"><span style="font-size:12.8px"
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg
gmail-m_-642682847458610453gmail_msg">Hello list,</span>
<div style="font-size:12.8px"
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg
gmail-m_-642682847458610453gmail_msg"><br
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg
gmail-m_-642682847458610453gmail_msg">
</div>
<div style="font-size:12.8px"
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg
gmail-m_-642682847458610453gmail_msg">I run one Erlang application on
Debian server and today I
find the beam process
consumes around 35G memory
by `top` command.</div>
<div style="font-size:12.8px"
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg
gmail-m_-642682847458610453gmail_msg"><br
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg
gmail-m_-642682847458610453gmail_msg">
</div>
<div style="font-size:12.8px"
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg
gmail-m_-642682847458610453gmail_msg">
<div
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg
gmail-m_-642682847458610453gmail_msg">```</div>
<div
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg
gmail-m_-642682847458610453gmail_msg">KiB Mem: 99194912 total, 61682656
used, 37512252 free,
397380 buffers</div>
<div
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg
gmail-m_-642682847458610453gmail_msg">KiB Swap: 0 total, 0
used, 0 free.
18684864 cached Mem</div>
<div
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg
gmail-m_-642682847458610453gmail_msg"><br
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg
gmail-m_-642682847458610453gmail_msg">
</div>
<div
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg
gmail-m_-642682847458610453gmail_msg"> PID USER PR NI VIRT
RES SHR S %CPU %MEM
TIME+ COMMAND</div>
<div
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg
gmail-m_-642682847458610453gmail_msg">11858 usr1 20 0 36.850g 0.032t
6220 S 73.5 34.4
8038:49 beam.smp</div>
<div
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg
gmail-m_-642682847458610453gmail_msg">```</div>
<div
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg
gmail-m_-642682847458610453gmail_msg"><br
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg
gmail-m_-642682847458610453gmail_msg">
</div>
<div
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg
gmail-m_-642682847458610453gmail_msg">I connect to the Erlang
application using remote
shell and find the
mem-leaked supervisor tree
and run gc on the whole
tree. Code looks like
blow:</div>
<div
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg
gmail-m_-642682847458610453gmail_msg"><br
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg
gmail-m_-642682847458610453gmail_msg">
</div>
<div
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg
gmail-m_-642682847458610453gmail_msg">```</div>
<div
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg
gmail-m_-642682847458610453gmail_msg">lists:foreach(fun(E) -> PId =
element(2, E),
erlang:garbage_collect(PId)
end,
supervisor:which_children(<wbr>some_thing_sup)).<br
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg
gmail-m_-642682847458610453gmail_msg">
</div>
<div
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg
gmail-m_-642682847458610453gmail_msg">```</div>
<div
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg
gmail-m_-642682847458610453gmail_msg"><br
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg
gmail-m_-642682847458610453gmail_msg">
</div>
<div
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg
gmail-m_-642682847458610453gmail_msg">and erlang:memory() decreases from
32G to 23G.</div>
<div
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg
gmail-m_-642682847458610453gmail_msg">```</div>
<div
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg
gmail-m_-642682847458610453gmail_msg">
<div
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg
gmail-m_-642682847458610453gmail_msg">[{total,22982011544},</div>
<div
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg
gmail-m_-642682847458610453gmail_msg"> {processes,12884182336},</div>
<div
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg
gmail-m_-642682847458610453gmail_msg"> {processes_used,12884170336},</div>
<div
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg
gmail-m_-642682847458610453gmail_msg"> {system,10097829208},</div>
<div
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg
gmail-m_-642682847458610453gmail_msg"> {atom,13828705},</div>
<div
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg
gmail-m_-642682847458610453gmail_msg"> {atom_used,13796692},</div>
<div
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg
gmail-m_-642682847458610453gmail_msg"> {binary,170530288},</div>
<div
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg
gmail-m_-642682847458610453gmail_msg"> {code,16450626},</div>
<div
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg
gmail-m_-642682847458610453gmail_msg"> {ets,9637717576}]</div>
</div>
<div
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg
gmail-m_-642682847458610453gmail_msg">```</div>
<div
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg
gmail-m_-642682847458610453gmail_msg"><br
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg
gmail-m_-642682847458610453gmail_msg">
</div>
<div
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg
gmail-m_-642682847458610453gmail_msg">However, when I input `top`
command, the beam process
still takes 35G memory.
What can I do to release
the 9G memory? Thanks</div>
<div
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg
gmail-m_-642682847458610453gmail_msg"><br
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg
gmail-m_-642682847458610453gmail_msg">
</div>
<div
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg
gmail-m_-642682847458610453gmail_msg">BR</div>
<div
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg
gmail-m_-642682847458610453gmail_msg">-Jack</div>
</div>
</div>
</div>
</div>
______________________________<wbr>_________________<br
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg
gmail-m_-642682847458610453gmail_msg">
erlang-questions mailing list<br
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg
gmail-m_-642682847458610453gmail_msg">
<a moz-do-not-send="true"
href="mailto:erlang-questions@erlang.org"
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg
gmail-m_-642682847458610453gmail_msg" target="_blank">erlang-questions@erlang.org</a><br
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg
gmail-m_-642682847458610453gmail_msg">
<a moz-do-not-send="true"
href="http://erlang.org/mailman/listinfo/erlang-questions"
rel="noreferrer"
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg
gmail-m_-642682847458610453gmail_msg" target="_blank">http://erlang.org/mailman/<wbr>listinfo/erlang-questions</a><br
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361m_4111392240134868422gmail_msg
gmail-m_-642682847458610453gmail_msg">
</blockquote>
</div>
</blockquote>
</div>
</div>
</div>
<div dir="ltr"
class="gmail-m_-642682847458610453gmail_msg">
<div class="gmail_extra
gmail-m_-642682847458610453gmail_msg">
<div
class="gmail-m_-642682847458610453gmail_msg"><br
class="gmail-m_-642682847458610453gmail_msg">
</div>
-- <br
class="gmail-m_-642682847458610453gmail_msg">
<div
class="gmail-m_-642682847458610453m_4539060469624083756m_-851902295025243361gmail_signature
gmail-m_-642682847458610453gmail_msg">Jack
Tang<br
class="gmail-m_-642682847458610453gmail_msg">
<br
class="gmail-m_-642682847458610453gmail_msg">
<br
class="gmail-m_-642682847458610453gmail_msg">
<a moz-do-not-send="true"
href="http://www.linkedin.com/in/jacktang"
class="gmail-m_-642682847458610453gmail_msg" target="_blank">http://www.linkedin.com/in/<wbr>jacktang</a><br
class="gmail-m_-642682847458610453gmail_msg">
</div>
</div>
</div>
______________________________<wbr>_________________<br
class="gmail-m_-642682847458610453gmail_msg">
erlang-questions mailing list<br
class="gmail-m_-642682847458610453gmail_msg">
<a moz-do-not-send="true"
href="mailto:erlang-questions@erlang.org"
class="gmail-m_-642682847458610453gmail_msg"
target="_blank">erlang-questions@erlang.org</a><br
class="gmail-m_-642682847458610453gmail_msg">
<a moz-do-not-send="true"
href="http://erlang.org/mailman/listinfo/erlang-questions"
rel="noreferrer"
class="gmail-m_-642682847458610453gmail_msg"
target="_blank">http://erlang.org/mailman/<wbr>listinfo/erlang-questions</a><br
class="gmail-m_-642682847458610453gmail_msg">
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div class="gmail_signature">Jack Tang<br>
<br>
<br>
<a moz-do-not-send="true"
href="http://www.linkedin.com/in/jacktang"
target="_blank">http://www.linkedin.com/in/jacktang</a><br>
</div>
</div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
erlang-questions mailing list
<a class="moz-txt-link-abbreviated" href="mailto:erlang-questions@erlang.org">erlang-questions@erlang.org</a>
<a class="moz-txt-link-freetext" href="http://erlang.org/mailman/listinfo/erlang-questions">http://erlang.org/mailman/listinfo/erlang-questions</a>
</pre>
</blockquote>
<br>
</body>
</html>