[erlang-questions] Efficient sum matrix by column

[Éric Pailleau]

Hi,

Did you try gb_trees instead ?
Iterator/1 is very efficient. 

Le 12 avr. 2015 21:08, Alexander Petrovsky <askjuise@REDACTED> a écrit :
>
> Hi!
>
> 2015-04-12 22:18 GMT+04:00 ILYA Khlopotov <ilya.khlopotov@REDACTED>:
>>
>> I would suggest to try couple of things.
>>
>> 1. If N is always the same and as you've mentioned contains same keys and the total number of elements in proplists is less than 67108863 you can replace proplist with tuple. In this case you can use either foldl with getelement/setelement.
>
>
> Yes, it's less then 67198863. Do I understand correctly, you propose instead prolists like [{a,1}, {b,2} ...] use tuple like {a, 1, b, 2, ...} or just {1, 2, ...} ? Does it will be more efficient(getelement/setelement) then pattern matching in list comprehension over proplist?
>  
>>
>> 2. review the requirements and consider incremental way of updating the counters on insertion.
>
>
> Yep, as I say in my first letter, I'm using incremental way right now via ets:update_counter, it takes about 1.5 secs when N=2. But it's still too slow. 
>  
>>
>>
>> BR,
>> ILYA
>>
>>
>>
>>
>> On Sun, Apr 12, 2015 at 10:53 AM, Grigory Fateyev <gfborn@REDACTED> wrote:
>>>
>>> Hello Alexander Petrovsky!
>>> On Sun, 12 Apr 2015 21:23:09 +0400 you wrote:
>>>
>>> > Hello!
>>> >
>>> > I have about 100 nodes in cluster, each node in cluster contains about
>>> > 100000 elements in proplist. All proplists have equal length, the same
>>> > keys, but different values. I need to get the sum values for every key
>>> > between all proplists. By example, it can be represented as matrix NxM
>>> > (N~=100, M~=100000), and I need to get the sum by column.
>>> >
>>> > The problem is that it's too slow, even when I use the hackish way
>>> > with ets:update_counter and N=2, it's take about 1.5 secs, If I make
>>> > in parallel, it's take about 2-3 seconds.
>>> >
>>> > How can I make it fast and efficient?
>>> >
>>>
>>> Look at lists:keyfind/4 It is much faster than proplists.
>>>
>>> --
>>> Best regards!
>>> gfborn [at] gmail [dot] com
>>> _______________________________________________
>>> erlang-questions mailing list
>>> erlang-questions@REDACTED
>>> http://erlang.org/mailman/listinfo/erlang-questions
>>
>>
>>
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-questions@REDACTED
>> http://erlang.org/mailman/listinfo/erlang-questions
>>
>
>
>
> -- 
> Петровский Александр / Alexander Petrovsky,
>
> Skype: askjuise
> Phone: +7 914 8 820 815
>