[erlang-questions] Simple perf testing using rpc:pmap

[Todd]

Hi everyone,

Longtime lurker here. I decided to take the summer off and do something I've wanted to do for years: learn Erlang.

Newbie questions to follow...

Similar to the thread on 'distribution links utilizing resources'... I've been looking at Erlang in a couple of different ways, and here are some graphs (and code) to look at...

I started by using Erlang as an execution framework for comparing the performance of some common sorting algorithms:
* source : https://github.com/ToddG/experimental/blob/master/interview-questions/glassdoor/sorting/erlang/sortz/src/sortz.erl
* results : https://github.com/ToddG/experimental/blob/master/interview-questions/glassdoor/sorting/erlang/sortz/docs/sorting_algorithms_analysis.ipynb

Then I took one of the algorithms and looked at the performance of this 'embarrassingly parallel' problem as it was spread across the available processors on my laptop via 'rpc:pmap':
* source : https://github.com/ToddG/experimental/blob/master/interview-questions/glassdoor/sorting/erlang/psortz/src/psortz.erl
* resoults : https://github.com/ToddG/experimental/blob/master/interview-questions/glassdoor/sorting/erlang/psortz/docs/parallel_sorting_analysis.ipynb

I was surprised a couple of things:
* It is ridiculously easy to use `rpc:pmap`
* However, the gains are mostly in adding a second or third processor, adding more processors yielded marginally better results, but really not worth the effort (IMHO)

I'd like to better understand the forces at work here:
Q: How to apply Amdahls Law here? I have enough data, I should be able to calculate the constant time factor...
Q: Why is the constant time factor so large? I expected much better overall performance when adding >3 processors...
Q: How does 'rpc:pmap' differ from the skel framework that Joe mentioned in this thread:
https://groups.google.com/forum/#!topic/erlang-programming/_smlcd6S9v0
https://skel.weebly.com/about-skel.html
Q: Does anyone have a write-up on how to measure performance of distributed Erlang/BEAM apps on linux? What tools are you using, what stats are the most relevant, what things are missing that you metric yourself, etc? Obviously there's a _mountain_ of tools and things to look at here... I've been looking into [sysstat (https://github.com/sysstat/sysstat](https://github.com/sysstat/sysstat)).

One of the first things I did when analyzing the algorithms in Part 1 was to start writing metrics to ETS...but then I discovered that Erlang already has tools like cover, cprof, fprof... In fact, it seems that whenever you are looking for something you need, it's actually there, somewhere.

-Todd Greenwood
toddg@REDACTED
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20190619/ea375f4c/attachment.htm>