[erlang-questions] Efficient sum matrix by column

[ILYA Khlopotov]

I would suggest to try couple of things.

1. If N is always the same and as you've mentioned contains same keys and
the total number of elements in proplists is less than 67108863 you can
replace proplist with tuple. In this case you can use either foldl with
getelement/setelement.
2. review the requirements and consider incremental way of updating the
counters on insertion.

BR,
ILYA




On Sun, Apr 12, 2015 at 10:53 AM, Grigory Fateyev <gfborn@REDACTED> wrote:

> Hello Alexander Petrovsky!
> On Sun, 12 Apr 2015 21:23:09 +0400 you wrote:
>
> > Hello!
> >
> > I have about 100 nodes in cluster, each node in cluster contains about
> > 100000 elements in proplist. All proplists have equal length, the same
> > keys, but different values. I need to get the sum values for every key
> > between all proplists. By example, it can be represented as matrix NxM
> > (N~=100, M~=100000), and I need to get the sum by column.
> >
> > The problem is that it's too slow, even when I use the hackish way
> > with ets:update_counter and N=2, it's take about 1.5 secs, If I make
> > in parallel, it's take about 2-3 seconds.
> >
> > How can I make it fast and efficient?
> >
>
> Look at lists:keyfind/4 It is much faster than proplists.
>
> --
> Best regards!
> gfborn [at] gmail [dot] com
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20150412/2d666053/attachment.htm>