[erlang-patches] Fix for: extremely slow Erlang to/from Java communication via JInterface

[Henrik Nord]

Hello

Thank you for the contribution!
If the tests pass, the patch is likely to be included in R15B.



On 11/30/2011 10:32 AM, Nico Kruber wrote:
> We discovered, that the communication between Erlang and Java is
> extremely slow when sending longer lists.
>
> We had two lists, each containing about **700.000** Strings (typically of
> length 20-30). This resulted in a message size of about **40MB** when being
> sent over to Erlang. The transformation of these lists into the (binary)
> message took Java about **10 minutes** in an RPC call.
>
> The reason for this is in the JInterface, which incrementally allocates memory
> for the send-buffer in OtpOutputStream and increases this buffer by a constant
> size of 2KB in each iteration. Each increase involves a complete copying of
> the old buffer to a new one - for this **40MB** message, about 20000 memory
> copies are needed!
>
> We can reduce the send time from 10 minutes to less than 10 seconds (almost
> two orders of magnitude) with the patch below. It allocates 50% more
> memory in each step, instead of constantly adding only 2k. (This behaviour was
> inspired by the one used by java.util.ArrayList.)
>
> The following table provides some examples of how many memory copies are
> needed in order to reach a certain size:
>
> size         n*2048   (n*3)/2+1 and at least +2048
>     1K             1              1
>     2K             1              1
>     5K             3              3
>    10K             5              5
>    20K            10              6
>   100K            50             10
>   500K           250             14
> 1024K=1M        512             16
>       10M       5120             22
>       50M      25600             26
>      100M      51200             28
>
>
> git fetch git://github.com/NicoK/otp.git jinterface_better_buffer_alloc
>
> https://github.com/NicoK/otp/compare/jinterface_better_buffer_alloc
> https://github.com/NicoK/otp/compare/jinterface_better_buffer_alloc.patch
>
>
> It would be nice to see this in R15B as it significantly improves performance
> for large messages as shown.
>
>
> Regards
> Nico Kruber
>
> Note: Sending binary data is not affected, as it already allocates as much
> memory as needed to include the bytes.
>
>
> _______________________________________________
> erlang-patches mailing list
> erlang-patches@REDACTED
> http://erlang.org/mailman/listinfo/erlang-patches

-- 
/Henrik Nord Erlang/OTP

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-patches/attachments/20111130/7b7f5d66/attachment.htm>