[erlang-questions] Ref counted binary in ETS

[Frank Muller]

Thank you Michael. Very appreciated details.

Frank.

Le Friday, October 28, 2016, Michael Truog <mjtruog@REDACTED> a écrit :

> Binaries are the only Erlang data type that is reference counted, so you
> either would be sticking with Erlang binaries in Erlang source code, or
> handling the data in a NIF or port driver to get quicker lookup times.
> However, Erlang's main advantage is the fault-tolerance features and you
> sabotage that with potentially unstable NIFs or port drivers, so it is best
> to keep your usage in Erlang source code.
>
> The other things to think about, is if you can use less data or less
> readers, just as a basic change (probably an algorithmic change).  The
> general approach that avoids changing the design is grouping the binaries
> into larger binaries though, as was suggested.
>
> Best Regards,
> Michael
>
> On 10/27/2016 09:13 PM, Frank Muller wrote:
>
> Michael,
>
> Thank you for sharing "blookup". I'll look at it today.
>
> Any other ideas to minimise the overhead when sharing large Bins? I'm ok
> with changing the overall program design if necessary.
>
> Frank.
>
> Le Friday, October 28, 2016, Michael Truog <mjtruog@REDACTED
> <javascript:_e(%7B%7D,'cvml','mjtruog@REDACTED');>> a écrit :
>
>> You could use something like https://github.com/okeuday/blookup/ to make
>> larger binaries, but ideally you don't have to (since it makes the lookups
>> slow).  This would be the concept of grouping keys and values together,
>> putting them into various buckets, to decrease the ETS usage but increase
>> the binary size.
>>
>> On 10/27/2016 03:25 PM, Tony Rogvall wrote:
>>
>> Simple, you may your 512 byte memory chunks bigger :-)
>>
>> /Tony
>>
>>
>> On 27 okt 2016, at 23:24, Frank Muller <frank.muller.erl@REDACTED> wrote:
>>
>> Tony,
>>
>> Wow, 48-bytes is a huge overhead as N ~= 1000, and Every reader reads a binary 10x times or so per second.
>>
>> Can anyone please suggest a better strategy to efficiently share them with a lesser overhead?
>>
>> Frank.
>>
>> Le Thursday, October 27, 2016, Tony Rogvall <tony@REDACTED> a écrit :
>> 1. A reference.
>>
>> It will however build a 6-word structure called ProcBin on the heap for every binary reference that you read.
>> On a 64 bit machine that will lead to a 48 byte overhead. Nearly 10%.
>>
>> /Tony
>>
>>
>> On 27 okt 2016, at 20:47, Frank Muller <frank.muller.erl@REDACTED> wrote:
>>
>> Hi guys
>>
>> Suppose an ETS table with only 1 writer and N readers
>> This table contains only large binaries (>64B) with the same size 512B exactly.
>>
>> The single writer update these Bins once a while.
>> The N readers regularly read one of these Bins, but never change/update them.
>>
>> Questions: what a reader gets when reading a Bin from this ETS table:
>>
>> 1. a reference to that Bin
>> 2. or a full copy of that Bin
>>
>> Hope someone can clarify this (with an example).
>>
>> Thank you
>> Frank
>>
>> _______________________________________________
>> erlang-questions mailing listerlang-questions@REDACTED://erlang.org/mailman/listinfo/erlang-questions
>>
>>
>>
>> _______________________________________________
>> erlang-questions mailing listerlang-questions@REDACTED://erlang.org/mailman/listinfo/erlang-questions
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20161027/5b7e73cb/attachment.htm>