[erlang-questions] Non-blocking BEAM code loading?

[Paolo Negri]

We run an application which runs thousands of long lived processes and
we see the system blocking on code purge during code updates.
I remember that Kenneth Lundin at the recent Erlang User Conference
announced that something related to code loading optimization is in
the erlang roadmap, hopefully slides will be published soon [1], if I
remember well the change was related to spreading code purge across
all the available cores while currently a single core is actually used
to perform the operation.

We also use the trick of compiling data in modules in order to push
data in the constant pool but we actually have thousands of small
terms (rendered as one function clause per term) and loading these
modules doesn't seem to block, but in our case I guess that the
overall size is much less than 60MB.

[1] http://www.erlang-factory.com/conference/ErlangUserConference2011/speakers/KennethLundin

Paolo

On Sun, Nov 6, 2011 at 5:02 AM, Bob Ippolito <bob@REDACTED> wrote:
> Normally just a few hundred, purge isn't the slow part for us and I don't
> believe that it blocks at all (not that I noticed).
>
> On Saturday, November 5, 2011, Robert Virding
> <robert.virding@REDACTED> wrote:
>> If you have many processes then code loading can take a noticeable time.
>> The code server must purge old versions of a module which it does by going
>> through all processes checking each one if it running the old code and if so
>> killing it. I don't know if this blocks all the schedulers and if so why,
>> but it can take a noticeable time to do.
>>
>> Robert
>>
>>
>> ________________________________
>>
>> ETS is no good for our use case, we have ~60MB worth of uncompressed
>> serialized terms (nested gb_trees mostly) that we need live in a given
>> request. We traverse it very quickly and end up with a very small list of
>> terms as the result (essentially a filter on a nested structure). A no-copy
>> ets would work, but since the work is so short lived and code is tightly
>> associated to this structure I think that our current solution is
>> appropriate as long as we can fix the blocking.
>>
>> "declare constant" may also work, but I think it is more practical to just
>> make code loading better in the short term (which has other benefits). You
>> could implement "declare constant" on top of the code loader, we have a
>> mochiglobal module in mochiweb that basically serves that purpose.
>>
>> Using a module is a convenient way to give concurrent access to the data
>> to hundreds of simultaneous processes with minimal serialization.
>>
>> -bob
>>
>> On Saturday, November 5, 2011, Björn-Egil Dahlberg
>> <wallentin.dahlberg@REDACTED> wrote:
>>> Yes, it is a simple (and currently only way) to push data to the constant
>>> pool. You could use ETS instead. It would of course also remove data from
>>> the heap and reduce GC copy strain but introduce copy on any read.
>>> Björn Gustavsson talked about introducing a "declare constant" function
>>> earlier but i don't know if he has done any work on it. The use case was the
>>> same as for you, pushing lookup structures from gb_trees and gb_sets. But,
>>> solving code loading would probably be a better prioritization.
>>> I would like to think that the garbage collector should solve this. Data
>>> sets which are read only and live are tenured to a generational heap and not
>>> included in minor gc phases. Putting it in a constant removes it all
>>> together of course but i would like the garbage collector to identify and
>>> handle this with generational strategies. The trade off is generational
>>> heaps linger and may hold dead data longer than necessary.
>>>
>>>
>>> Den 5 november 2011 21:30 skrev Bob Ippolito <bob@REDACTED>:
>>>>
>>>> We abuse code loading "upgrades" so that we can share memory and reduce
>>>> GC pressure for large data structures that do not change quickly (once every
>>>> few minutes). Works great except for all the blocking!
>>>>
>>>> On Saturday, November 5, 2011, Björn-Egil Dahlberg
>>>> <wallentin.dahlberg@REDACTED> wrote:
>>>> > There is no other locking for code loading than blocking. This is an
>>>> > optimization of course since locking mechanism overhead is removed from the
>>>> > equation. Code loading is not used all that often in the normal cases
>>>> > besides startups and upgrades.
>>>> > That being said, there are plans to remove this "stop-the-world"
>>>> > strategy since it is blocking other strategies and optimizations. Also, we
>>>> > are well aware of that blocking does degrade performance when loading new
>>>> > modules and does not agree with our concurrency policy.
>>>> > I think we can lessen the time blocked in the current implementation
>>>> > but the blocking strategy should (and probably will) be removed. Nothing
>>>> > planned as of yet though.
>>>> > Regards,
>>>> > Björn-Egil
>>>> >
>>>> > 2011/11/5 Bob Ippolito <bob@REDACTED>
>>>> >>
>>>> >> We've found a bottleneck in some of our systems, when we load in
>>>> >> large
>>>> >> new modules there is a noticeable pause (1+ seconds) that blocks all
>>>> >> of the schedulers. It looks like this is because the
>>>> >> erlang:load_binary/2 BIF blocks SMP before it does anything at all.
>>>> >>
>>>> >> It would be a big win for us if more of this happened without
>>>> >> blocking
>>>> >> the VM, there's a lot of busy work in loading a module that shouldn't
>>>> >> need any locking. For example, decompressing and decoding the literal
>>>> >> table is probably where our code spends almost all of its time.
>>>> >>
>>>> >> There aren't a lot of comments for why it needs to lock the VM,
>>>> >> especially for the whole of load_binary. Are there any hidden gotchas
>>>> >> in here that I should know about before giving it a try? I'm unable
>>>> >> to
>>>> >> find much where the block is actually necessary, but I am not very
>>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
>



-- 
Engineering
http://www.wooga.com | phone +49-30-8962 5058  | fax +49-30-8964 9064

wooga GmbH | Saarbruecker Str. 38 | 10405 Berlin | Germany
Sitz der Gesellschaft: Berlin; HRB 117846 B
Registergericht Berlin-Charlottenburg
Geschaeftsfuehrung: Jens Begemann, Philipp Moeser