[erlang-questions] Strange performance degradation in dict when it's storing lists

[Park, Sungjin]

@Sverker, you are right.  Still I'm curious what makes the big difference.
There was no such big difference when the values were tuples rather than
lists.  The extra overhead you mentioned must be the same for both.

@Kyungho, I tested in otp-18.  Can there possibly be any VM change related
with this?



On Tue, Aug 23, 2016 at 5:01 AM, Kyungho Yun <inanun@REDACTED> wrote:

> Hi there.
>
> The various size of return values could be confusable.
> So I tested original source and some modified source; value() ->
> lists:seq(0,255),
> though I couldn't get such a big value in OTP 19.
>
> But I guess GC might be the cause.
>
> and the reference of binary may use for messaging in same node.
>
>
>
> On Mon, Aug 22, 2016 at 10:53 PM, Sverker Eriksson <
> sverker.eriksson@REDACTED> wrote:
>
>> Looks like you measure the latency of send + dict:find + receive
>> that sometimes happens to get preemted by very heavy calls
>> to lists:delete.
>>
>> /Sverker
>>
>>
>>
>> On 08/22/2016 10:35 AM, Park, Sungjin wrote:
>>
>> I observed a strange performance degradation in dict.  Let me share the
>> code I used in the test first.
>>
>>
>>     -module(data).
>>     -export([start_link/1, get/1, get_concurrent/1]).
>>     -export([init/0]).
>>
>>     start_link() ->
>>       proc_lib:start_link(?MODULE, init, []).
>>
>>     init() ->
>>       register(?MODULE, self()),
>>       % Initialize data:
>>       % 0 => [],
>>       % 1 => [1],
>>       % 2 => [1,2]
>>       % ...
>>       Dict = lists:foldl(
>>         fun (Key, Dict0) -> dict:store(Key, value(Key), Dict0) end,
>>         dict:new(), lists:seq(0, 255)
>>       ),
>>       proc_lib:init_ack({ok, self()}),
>>       loop(Dict).
>>
>>     value(Key) ->
>>         lists:seq(1, Key).
>>
>>     loop(Dict) ->
>>       receive
>>         {get, Key, From} ->
>>           case dict:find(Key, Dict) of
>>             {ok, Value} -> From ! Value;
>>             error -> From ! undefined
>>           end;
>>         _ ->
>>           ok
>>       end,
>>       loop(Dict).
>>
>>     get(Key) ->
>>       ?MODULE ! {get, Key, self()},
>>       receive
>>         Value -> Value
>>       end.
>>
>>     %% Run get N times and return average execution time.
>>     -spec get_concurrent(integer()) -> number().
>>     get_concurrent(N) ->
>>       Profiler = self(),
>>       Workers = [
>>          prof_lib:spawn_link(
>>            fun () ->
>>              Key = erlang:system_time() rem 255,
>>              Result = timer:tc(?MODULE, get, [Key]),
>>              Profiler ! {self(), Result}
>>            end
>>          ) || _ <- lists:seq(1, N)
>>        ],
>>        Ts = receive_all(Workers, []),
>>        lists:sum(Ts) / length(Ts).
>>
>>     receive_all([], Ts) ->
>>       Ts;
>>     receive_all(Workers, Ts) ->
>>       receive
>>         {Worker, {T, _}} -> receive_all(lists:delete(Worker, Workers), [T |
>> Ts])
>>       end.
>>
>>
>> When I ran the test in the shell, I got.
>>
>>     1> data:start_link().
>>     {ok, <0.6497.46>}
>>     2> timer:tc(data, get, [5]).
>>     {23,[1,2,3,4,5]}
>>
>>
>> I could get a value in 23 microseconds and expected something not too
>> slower results for concurrent get but,
>>
>>     3> data:get_concurrent(100000).
>>     19442.828
>>
>>
>> The value 19442.828 microseconds seemed to be too big a value so I tested
>> with different values such as large binaries and tuples.  And this time the
>> same get_concurrent(100000) gave me 200 something microseconds.
>>
>> I also tried the same with an ets instead of a dict, but there was no such
>> performance degradation by the value type.
>>
>>
>>
>>
>> _______________________________________________
>> erlang-questions mailing listerlang-questions@REDACTED://erlang.org/mailman/listinfo/erlang-questions
>>
>>
>>
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-questions@REDACTED
>> http://erlang.org/mailman/listinfo/erlang-questions
>>
>>
>


-- 
Park, Sungjin
-------------------------------------------------------------------------------------------------------------------
Peculiar travel suggestions are dancing lessons from god.
  -- The Books of Bokonon
-------------------------------------------------------------------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20160823/499bda95/attachment.htm>