<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Hello<br>
<br>
Thank you for the contribution!<br>
If the tests pass, the patch is likely to be included in R15B.<br>
<br>
<br>
<br>
On 11/30/2011 10:32 AM, Nico Kruber wrote:
<blockquote cite="mid:10958143.tlbGcITqxT@csr-pc40.zib.de"
type="cite">
<pre wrap="">We discovered, that the communication between Erlang and Java is
extremely slow when sending longer lists.
We had two lists, each containing about **700.000** Strings (typically of
length 20-30). This resulted in a message size of about **40MB** when being
sent over to Erlang. The transformation of these lists into the (binary)
message took Java about **10 minutes** in an RPC call.
The reason for this is in the JInterface, which incrementally allocates memory
for the send-buffer in OtpOutputStream and increases this buffer by a constant
size of 2KB in each iteration. Each increase involves a complete copying of
the old buffer to a new one - for this **40MB** message, about 20000 memory
copies are needed!
We can reduce the send time from 10 minutes to less than 10 seconds (almost
two orders of magnitude) with the patch below. It allocates 50% more
memory in each step, instead of constantly adding only 2k. (This behaviour was
inspired by the one used by java.util.ArrayList.)
The following table provides some examples of how many memory copies are
needed in order to reach a certain size:
size n*2048 (n*3)/2+1 and at least +2048
1K 1 1
2K 1 1
5K 3 3
10K 5 5
20K 10 6
100K 50 10
500K 250 14
1024K=1M 512 16
10M 5120 22
50M 25600 26
100M 51200 28
git fetch git://github.com/NicoK/otp.git jinterface_better_buffer_alloc
<a class="moz-txt-link-freetext" href="https://github.com/NicoK/otp/compare/jinterface_better_buffer_alloc">https://github.com/NicoK/otp/compare/jinterface_better_buffer_alloc</a>
<a class="moz-txt-link-freetext" href="https://github.com/NicoK/otp/compare/jinterface_better_buffer_alloc.patch">https://github.com/NicoK/otp/compare/jinterface_better_buffer_alloc.patch</a>
It would be nice to see this in R15B as it significantly improves performance
for large messages as shown.
Regards
Nico Kruber
Note: Sending binary data is not affected, as it already allocates as much
memory as needed to include the bytes.</pre>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
erlang-patches mailing list
<a class="moz-txt-link-abbreviated" href="mailto:erlang-patches@erlang.org">erlang-patches@erlang.org</a>
<a class="moz-txt-link-freetext" href="http://erlang.org/mailman/listinfo/erlang-patches">http://erlang.org/mailman/listinfo/erlang-patches</a></pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
/Henrik Nord Erlang/OTP</pre>
</body>
</html>