[erlang-questions] investigating memory issues

[Jesper Louis Andersen]

The latter, ...

If you have some JSON and you decode that into, say,

#{
    <<"key">> := SomeValue,
    ...
}

Then SomeValue is a subbinary which keeps JSON alive. Now, what to do
depends on the situation:

1. If SomeValue is not long-lived and you are throwing JSON away roughly at
the same time as SomeValue, then you have no problems right away.

2. If SomeValue is long-lived and JSON is not, then you could potentially
keep JSON around for a long time. This is where you can do X =
binary:copy(SomeValue) to force X to be a fresh copy. This then puts you
back into situation 1, and you will not have a problem.

It is not a priori obvious what is the right thing to do. There are
trade-offs with processing speed, ease of use, gc behaviour and so on. If,
for instance, jsonx made a new binary itself, then it would lose every
performance benchmark to an implementation which uses the subbinary
construction. And people tend to value their decoding speed over the
ability to take out a VM for some odd reason :)


On Mon, Oct 19, 2015 at 1:33 PM, Alexander Petrovsky <askjuise@REDACTED>
wrote:

> If I understand correctly, you suggest call X = binary:copy(SomeBinary)
> and then call jsonx parser with X as parameter? Or just going through
> parsed result and then call binary:copy over each binary element?
>
> 2015-10-17 13:50 GMT+03:00 Jesper Louis Andersen <
> jesper.louis.andersen@REDACTED>:
>
>>
>> On Fri, Oct 16, 2015 at 9:34 PM, Caragea Silviu <silviu.cpp@REDACTED>
>> wrote:
>>
>>> 7. jsonx
>>
>>
>> This is a shot but I think this library is your problem. When it parses a
>> JSON document, it uses enif_make_sub_binary on the binary() containing the
>> JSON string. This means if you do not use binary:copy() on data you hoist
>> out of the JSON document, then you are keeping the whole document around
>> for as long as you refer to that subbinary.
>>
>> Try wrapping strings you hoist out in binary:copy/1. This will slow you
>> down, but it should remove the reference to the original binary which would
>> make your system be able to reclaim that memory. Alternatively, if your
>> system is not highly loaded, you could simply replace jsonx for jsx which
>> is written in Erlang and avoids such problems (at the expense of slower
>> parsing, but JSON will *never* win a parsing race anyway and it is better
>> to swtch the format then).
>>
>> I've seen systems stuffing such subbinaries into ETS, and then your ETS
>> table is keeping data around.
>>
>>
>> --
>> J.
>>
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-questions@REDACTED
>> http://erlang.org/mailman/listinfo/erlang-questions
>>
>>
>
>
> --
> Петровский Александр / Alexander Petrovsky,
>
> Skype: askjuise
> Phone: +7 914 8 820 815
>
>


-- 
J.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20151019/984b7b47/attachment.htm>