[erlang-questions] Non-blocking BEAM code loading?

Bob Ippolito bob@REDACTED
Sat Nov 5 18:40:01 CET 2011


We've found a bottleneck in some of our systems, when we load in large
new modules there is a noticeable pause (1+ seconds) that blocks all
of the schedulers. It looks like this is because the
erlang:load_binary/2 BIF blocks SMP before it does anything at all.

It would be a big win for us if more of this happened without blocking
the VM, there's a lot of busy work in loading a module that shouldn't
need any locking. For example, decompressing and decoding the literal
table is probably where our code spends almost all of its time.

There aren't a lot of comments for why it needs to lock the VM,
especially for the whole of load_binary. Are there any hidden gotchas
in here that I should know about before giving it a try? I'm unable to
find much where the block is actually necessary, but I am not very
familiar with the BEAM implementation yet.

I expect that the erts_export_consolidate, insert_new_code and
final_touch are really the only things that need so much
serialization, and maybe the set_default_trace_pattern… is there
anything big that I'm missing? It seems that breaking up
erts_load_module into two functions (one to do all the decoding
without the erts_smp_block_system(0), and the other to do the
integration work with the block) would be straightforward.

-bob



More information about the erlang-questions mailing list