[erlang-questions] ETS and CPU

[Robert Virding]

As others have said that this will copy the map from the spawner to the
spawnee process. The interesting thing is what you do with afterwards. If
you just carry it around as process local data then there is no extra
copying except during GC which you can't do anything about. If, however,
you then store the map as a map in an ETS table you will be copying the map
between the process and the table for every access. This will be very
inefficient if you just want to access one key in the map as you will first
have to copy the whole map from the ETS table to the process, access the
key, and then if you are writing to the map copy it all back to the ETS
table. It would then be better to split the map into separate {Key,Value}
tuples which you store as separate elements in the ETS table.

ETS tables are great but you have to be aware of the copying. Erlang
doesn't do shared global data! ETS tables aren't shared, just globally
accessible.

Robert


On 15 March 2016 at 16:07, Alex Howle <itshowlertime@REDACTED> wrote:

> The map is not being sent as a message. It is passed in to the spawned
> processes by being in scope of the spawned function.
>
> Pseudocode:
>
> A=bigmap,
> spawn(fun()-> do_something(A) end).
> On 15 Mar 2016 13:43, "Alex Howle" <itshowlertime@REDACTED> wrote:
>
>> The map is not being sent as a message. It is passed in to the spawned
>> processes by being in scope of the spawned function.
>>
>> Pseudocode:
>>
>> A=bigmap,
>> spawn(fun()-> do_something(A) end).
>> On 15 Mar 2016 11:32, "Sverker Eriksson" <sverker.eriksson@REDACTED>
>> wrote:
>>
>>> Each successful ets:lookup call is a copy operation of the entire term
>>> from ETS to the process heap.
>>>
>>> If you are comparing ets:lookup of big map
>>> to sending big map in message then I would expect
>>> ets:lookup to win, as copy_shallow (used by ets:lookup)
>>> is optimized to be faster than copy_struct (used by send).
>>>
>>>
>>> /Sverker, Erlang/OTP
>>>
>>>
>>> On 03/15/2016 09:52 AM, Alex Howle wrote:
>>>
>>> I've been experiencing an issue and was wondering if anyone else has any
>>> experience in this area. I've stripped back the problem to its bare bones
>>> for the purposes of this mail.
>>>
>>>
>>>
>>> I have an Erlang 18.1 application that uses ETS to store an Erlang map
>>> structure. Using erts_debug:flat_size/1 I can approximate the map's size to
>>> be 1MB. Upon the necessary activity trigger the application spawns about 25
>>> short-lived processes to perform the main work of the application. This
>>> activity trigger is fired roughly 9 times a second under normal operating
>>> conditions. Each of these 25 processes performs 1 x ets:lookup/2 calls to
>>> read from the map.
>>>
>>>
>>>
>>> What I've found is that the above implementation has a CPU profile that
>>> is quite "expensive" - each of the CPU cores (40 total comprised of 2
>>> Processors with 10 hyperthreaded cores) frequently runs at 100%. The
>>> machine in question also has 32GB RAM of which about 9GB is used at peak.
>>> There is no swap usage whatsoever. Examination shows that copy_shallow is
>>> performing the most work.
>>>
>>>
>>>
>>> After changing the implementation so that the 25 spawned processes no
>>> longer read from the ETS table to retrieve the map structure and, instead
>>> the map is passed to the processes on spawn, the CPU usage on the server is
>>> considerably lower.
>>>
>>>
>>>
>>> Can anyone offer advice as to why I'm seeing the differing CPU profiles?
>>>
>>>
>>> _______________________________________________
>>> erlang-questions mailing listerlang-questions@REDACTED://erlang.org/mailman/listinfo/erlang-questions
>>>
>>>
>>>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20160316/7a6578bf/attachment.htm>