[erlang-questions] Efficient sum matrix by column

[Alexander Petrovsky]

The proplists already sorted. Thanks, I will look at N-ary tree. Looks like
the NIF it's not a good idea due the sum of two lists with 100000 each will
be more then 1ms (btw I know about dirty schedulers).

2015-04-12 22:42 GMT+04:00 Hynek Vychodil <vychodil.hynek@REDACTED>:

> Sort by proper key and then merge join. Form N-nary tree between your
> nodes for merging and experiment with N. If not fast enough, convert to
> binaries and write NIFs, compress (Snappy, LZ4, ...) and so on. Unless you
> write more how keys in your proplists look like, there is not so much to
> advice.
>
> On Sun, Apr 12, 2015 at 7:23 PM, Alexander Petrovsky <askjuise@REDACTED>
> wrote:
>
>> Hello!
>>
>> I have about 100 nodes in cluster, each node in cluster contains about
>> 100000 elements in proplist. All proplists have equal length, the same
>> keys, but different values. I need to get the sum values for every key
>> between all proplists. By example, it can be represented as matrix NxM
>> (N~=100, M~=100000), and I need to get the sum by column.
>>
>> The problem is that it's too slow, even when I use the hackish way with
>> ets:update_counter and N=2, it's take about 1.5 secs, If I make in
>> parallel, it's take about 2-3 seconds.
>>
>> How can I make it fast and efficient?
>>
>> --
>> Петровский Александр / Alexander Petrovsky,
>>
>> Skype: askjuise
>> Phone: +7 914 8 820 815
>>
>>
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-questions@REDACTED
>> http://erlang.org/mailman/listinfo/erlang-questions
>>
>>
>


-- 
Петровский Александр / Alexander Petrovsky,

Skype: askjuise
Phone: +7 914 8 820 815
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20150412/eb014fab/attachment.htm>