[erlang-questions] Non-blocking BEAM code loading?

[Robert Virding]

If you have many processes then code loading can take a noticeable time. The code server must purge old versions of a module which it does by going through all processes checking each one if it running the old code and if so killing it. I don't know if this blocks all the schedulers and if so why, but it can take a noticeable time to do. 

Robert 

----- Original Message -----

> ETS is no good for our use case, we have ~60MB worth of uncompressed
> serialized terms (nested gb_trees mostly) that we need live in a
> given request. We traverse it very quickly and end up with a very
> small list of terms as the result (essentially a filter on a nested
> structure). A no-copy ets would work, but since the work is so short
> lived and code is tightly associated to this structure I think that
> our current solution is appropriate as long as we can fix the
> blocking.

> "declare constant" may also work, but I think it is more practical to
> just make code loading better in the short term (which has other
> benefits). You could implement "declare constant" on top of the code
> loader, we have a mochiglobal module in mochiweb that basically
> serves that purpose.

> Using a module is a convenient way to give concurrent access to the
> data to hundreds of simultaneous processes with minimal
> serialization.

> -bob

> On Saturday, November 5, 2011, Björn-Egil Dahlberg <
> wallentin.dahlberg@REDACTED > wrote:
> > Yes, it is a simple (and currently only way) to push data to the
> > constant pool. You could use ETS instead. It would of course also
> > remove data from the heap and reduce GC copy strain but introduce
> > copy on any read.
> > Björn Gustavsson talked about introducing a "declare constant"
> > function earlier but i don't know if he has done any work on it.
> > The use case was the same as for you, pushing lookup structures
> > from gb_trees and gb_sets. But, solving code loading would
> > probably be a better prioritization.
> > I would like to think that the garbage collector should solve this.
> > Data sets which are read only and live are tenured to a
> > generational heap and not included in minor gc phases. Putting it
> > in a constant removes it all together of course but i would like
> > the garbage collector to identify and handle this with
> > generational strategies. The trade off is generational heaps
> > linger and may hold dead data longer than necessary.
> >
> >
> > Den 5 november 2011 21:30 skrev Bob Ippolito < bob@REDACTED >:
> >>
> >> We abuse code loading "upgrades" so that we can share memory and
> >> reduce GC pressure for large data structures that do not change
> >> quickly (once every few minutes). Works great except for all the
> >> blocking!
> >>
> >> On Saturday, November 5, 2011, Björn-Egil Dahlberg <
> >> wallentin.dahlberg@REDACTED > wrote:
> >> > There is no other locking for code loading than blocking. This
> >> > is an optimization of course since locking mechanism overhead
> >> > is removed from the equation. Code loading is not used all that
> >> > often in the normal cases besides startups and upgrades.
> >> > That being said, there are plans to remove this "stop-the-world"
> >> > strategy since it is blocking other strategies and
> >> > optimizations. Also, we are well aware of that blocking does
> >> > degrade performance when loading new modules and does not agree
> >> > with our concurrency policy.
> >> > I think we can lessen the time blocked in the current
> >> > implementation but the blocking strategy should (and probably
> >> > will) be removed. Nothing planned as of yet though.
> >> > Regards,
> >> > Björn-Egil
> >> >
> >> > 2011/11/5 Bob Ippolito < bob@REDACTED >
> >> >>
> >> >> We've found a bottleneck in some of our systems, when we load
> >> >> in large
> >> >> new modules there is a noticeable pause (1+ seconds) that
> >> >> blocks all
> >> >> of the schedulers. It looks like this is because the
> >> >> erlang:load_binary/2 BIF blocks SMP before it does anything at
> >> >> all.
> >> >>
> >> >> It would be a big win for us if more of this happened without
> >> >> blocking
> >> >> the VM, there's a lot of busy work in loading a module that
> >> >> shouldn't
> >> >> need any locking. For example, decompressing and decoding the
> >> >> literal
> >> >> table is probably where our code spends almost all of its time.
> >> >>
> >> >> There aren't a lot of comments for why it needs to lock the VM,
> >> >> especially for the whole of load_binary. Are there any hidden
> >> >> gotchas
> >> >> in here that I should know about before giving it a try? I'm
> >> >> unable to
> >> >> find much where the block is actually necessary, but I am not
> >> >> very
> >> >> familiar with the BEAM implementation yet.
> >> >>
> >> >> I expect that the erts_export_consolidate, insert_new_code and
> >> >> final_touch are really the only things that need so much
> >> >> serialization, and maybe the set_default_trace_pattern… is
> >> >> there
> >> >> anything big that I'm missing? It seems that breaking up
> >> >> erts_load_module into two functions (one to do all the decoding
> >> >> without the erts_smp_block_system(0), and the other to do the
> >> >> integration work with the block) would be straightforward.
> >> >>
> >> >> -bob
> >> >> _______________________________________________
> >> >> erlang-questions mailing list
> >> >> erlang-questions@REDACTED
> >> >> http://erlang.org/mailman/listinfo/erlang-questions
> >> >
> >> >
> >
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20111106/87c4771d/attachment.htm>