[erlang-questions] High lock contention on dist_tables

[Brian Picciano]

Thanks for the heads up Lukas! Sorry I stopped responding, we ended up
solving the problem (for now) by drastically cutting down on inter-node
communication in another way, and this thread got lost in my inbox, but I
really appreciate the follow-up!


On Thu, May 16, 2013 at 3:43 AM, Lukas Larsson <lukas@REDACTED> wrote:

> Hello Brian,
>
> Just letting you know that I have just merged a fix which changes the
> rwlock I mentioned before to an rlock. This should reduce the contention
> which you are seeing if it was caused by many calls to erlang:nodes().
>
> Lukas
>
>
> On Tue, Apr 23, 2013 at 9:01 PM, Scott Lystig Fritchie <
> fritchie@REDACTED> wrote:
>
>> Brian Picciano <mediocregopher@REDACTED> wrote:
>>
>> bp> We have a pool of 3 erlang nodes, all on different servers. Every
>> bp> afternoon, without fail, we start seeing lots of messages between
>> bp> the nodes start having really high latency, on the order of tens of
>> bp> seconds. [...]
>>
>> Brian, it's probably worthwhile to continue chasing the 'lcnt' avenue
>> as you've been corresponding with Lukas...
>>
>> ... but at the same time, I also wonder about "tens of seconds".  My gut
>> says that such delays would require some amazingly high lock contention
>> rates.  Something that can cause such messaging delays much more easily
>> is network congestion/packet loss that triggers TCP slow start.  Many
>> Linux kernels have the RTO_min value at one second, which is the amount
>> of time to wait before entering slow start state.
>>
>> If network packet loss is a problem, this blog posting can explain one
>> reason why it's happening:
>> http://www.snookles.com/slf-blog/2012/01/05/tcp-incast-what-is-it/
>>
>> -Scott
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-questions@REDACTED
>> http://erlang.org/mailman/listinfo/erlang-questions
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20130612/e017a351/attachment.htm>