[erlang-questions] ETS and CPU

[Sverker Eriksson]

Well, I would expect copy_shallow (from ETS) to be less CPU intensive
than copy_struct (from process).

However, as indicated by others, ets:lookup on such a big map will probably
trigger a garbage collection on the process, which will lead to
yet another copy of the big map.

The spawn(fun() -> do_something(BigMap) end) on the other hand will
allocate a big enough heap for the process form the start and only do
one copy of the big map.

/Sverker, Erlang/OTP


On 03/16/2016 10:43 AM, Alex Howle wrote:
>
> Assuming that when you say "win" you mean that ets:lookup should be 
> more efficient (and less CPU intensive) then I'm seeing the opposite.
>
> On 15 Mar 2016 11:32, "Sverker Eriksson" 
> <sverker.eriksson@REDACTED <mailto:sverker.eriksson@REDACTED>> 
> wrote:
>
>     Each successful ets:lookup call is a copy operation of the entire term
>     from ETS to the process heap.
>
>     If you are comparing ets:lookup of big map
>     to sending big map in message then I would expect
>     ets:lookup to win, as copy_shallow (used by ets:lookup)
>     is optimized to be faster than copy_struct (used by send).
>
>
>     /Sverker, Erlang/OTP
>
>
>     On 03/15/2016 09:52 AM, Alex Howle wrote:
>>
>>     I've been experiencing an issue and was wondering if anyone else
>>     has any experience in this area. I've stripped back the problem
>>     to its bare bones for the purposes of this mail.
>>
>>     I have an Erlang 18.1 application that uses ETS to store an
>>     Erlang map structure. Using erts_debug:flat_size/1 I can
>>     approximate the map's size to be 1MB. Upon the necessary activity
>>     trigger the application spawns about 25 short-lived processes to
>>     perform the main work of the application. This activity trigger
>>     is fired roughly 9 times a second under normal operating
>>     conditions. Each of these 25 processes performs 1 x ets:lookup/2
>>     calls to read from the map.
>>
>>     What I've found is that the above implementation has a CPU
>>     profile that is quite "expensive" - each of the CPU cores (40
>>     total comprised of 2 Processors with 10 hyperthreaded cores)
>>     frequently runs at 100%. The machine in question also has 32GB
>>     RAM of which about 9GB is used at peak. There is no swap usage
>>     whatsoever. Examination shows that copy_shallow is performing the
>>     most work.
>>
>>     After changing the implementation so that the 25 spawned
>>     processes no longer read from the ETS table to retrieve the map
>>     structure and, instead the map is passed to the processes on
>>     spawn, the CPU usage on the server is considerably lower.
>>
>>     Can anyone offer advice as to why I'm seeing the differing CPU
>>     profiles?
>>
>>
>>
>>     _______________________________________________
>>     erlang-questions mailing list
>>     erlang-questions@REDACTED <mailto:erlang-questions@REDACTED>
>>     http://erlang.org/mailman/listinfo/erlang-questions
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20160316/ac354e9f/attachment.htm>