[erlang-questions] Efficient sum matrix by column

Alexander Petrovsky askjuise@REDACTED
Sun Apr 12 21:08:30 CEST 2015


Hi!

2015-04-12 22:18 GMT+04:00 ILYA Khlopotov <ilya.khlopotov@REDACTED>:

> I would suggest to try couple of things.
>
> 1. If N is always the same and as you've mentioned contains same keys and
> the total number of elements in proplists is less than 67108863 you can
> replace proplist with tuple. In this case you can use either foldl with
> getelement/setelement.
>

Yes, it's less then 67198863. Do I understand correctly, you propose
instead prolists like [{a,1}, {b,2} ...] use tuple like {a, 1, b, 2, ...}
or just {1, 2, ...} ? Does it will be more efficient(getelement/setelement)
then pattern matching in list comprehension over proplist?


> 2. review the requirements and consider incremental way of updating the
> counters on insertion.
>

Yep, as I say in my first letter, I'm using incremental way right now via
ets:update_counter, it takes about 1.5 secs when N=2. But it's still too
slow.


>
> BR,
> ILYA
>
>
>
>
> On Sun, Apr 12, 2015 at 10:53 AM, Grigory Fateyev <gfborn@REDACTED>
> wrote:
>
>> Hello Alexander Petrovsky!
>> On Sun, 12 Apr 2015 21:23:09 +0400 you wrote:
>>
>> > Hello!
>> >
>> > I have about 100 nodes in cluster, each node in cluster contains about
>> > 100000 elements in proplist. All proplists have equal length, the same
>> > keys, but different values. I need to get the sum values for every key
>> > between all proplists. By example, it can be represented as matrix NxM
>> > (N~=100, M~=100000), and I need to get the sum by column.
>> >
>> > The problem is that it's too slow, even when I use the hackish way
>> > with ets:update_counter and N=2, it's take about 1.5 secs, If I make
>> > in parallel, it's take about 2-3 seconds.
>> >
>> > How can I make it fast and efficient?
>> >
>>
>> Look at lists:keyfind/4 It is much faster than proplists.
>>
>> --
>> Best regards!
>> gfborn [at] gmail [dot] com
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-questions@REDACTED
>> http://erlang.org/mailman/listinfo/erlang-questions
>>
>
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
>


-- 
Петровский Александр / Alexander Petrovsky,

Skype: askjuise
Phone: +7 914 8 820 815
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20150412/68914a44/attachment.htm>


More information about the erlang-questions mailing list