[erlang-questions] Game of Erlang life [was: Load testing in parallel]

[Serge Aleynikov]

So far the best resource of OTP documentation is the on-line reference: 
http://www.erlang.org/doc.

Sometimes this documentation may seem odd or vague for newcomers, yet 
the more you learn Erlang/OTP the more you understand and appreciate the 
format, depth and value of the content.

Once you mastered first "Programming Erlang" chapters (ch.1-15, 
excluding OTP ones) I suggest that you start by studying on-line module 
documentation under "Basic Applications": stdlib and kernel.  They 
expose a wealth of functions, many of which you'll find used in all 
projects.  Despite the fact that it may seem like a lot of information, 
if you approach this studying methodically and patiently, you'll be able 
to filter content on the module level first (i.e. gaining high-level 
understanding of what each module does) and then in those modules that 
seem to trigger more of your interest (like kernel/erlang, stdlib/io, 
stdlib/ets, etc.) study documented functions.

Don't worry about OTP (aside from stdlib and kernel applications) as 
much in the beginning - when you feel you have a good grasp on basic 
functions available in the two apps above, read the Design Principles 
guide [1] to understand OTP behaviors - which is the "meat" part of the 
framework that gives you the power of increasing your coding 
productivity by implying practical and reliable abstractions.

At this point, go back to "Programming Erlang" and study the OTP-related 
chapters (ch.16 onward) that give you application examples of how to use 
  behaviors.  This will conclude your "first circle of Erlang life" and 
give you enough knowledge to move on to exploiting the power of 
Erlang/OTP in your projects.

The "second circle of Erlang life" will begin when your curiosity will 
drive you to understanding that Erlang is *open source* and you'll start 
exploring the sources of OTP applications included in distribution. 
That is when you'll actually begin seeing that on-line documentation is 
rather good.  When you are comfortable with OTP code you may find bugs 
and post them to the erlang-bugs@REDACTED list, and also make some 
contributions.

The "third circle of Erlang life" will begin when you discover that 
Erlang's performance sucks at number-crunching applications (which are 
not the domain of problems Erlang was designed for), and begin writing 
ports and drivers to overcome these bottlenecks.  Additionally As a 
conclusion of this circle you'll be quite fascinated at how much thought 
was put in preventing mis-behaving user code crash the Erlang emulator.

The "forth circle of Erlang life" will begin when your curiosity drives 
you to studying sources of the Erlang emulator.  Here you'll learn how 
I/O multiplexing, garbage collection, bytecode execution, etc. is done, 
and answer some of more advanced questions you might have accumulated 
about features of the run-time system.

By this time others will find that you are "... more object oriented in
thinking than most of his C++ guys" [2]. The "next circle of Erlang 
life" will begin when you manage to get a job that will let your 
evolution as an Erlang programmer use acquired knowledge during 
day-time.  Since not many companies know the hidden powers of Erlang, 
this will largely depend on your abilities to show the strength of 
Erlang/OTP development to your manager, find suitable projects, and 
generate spin to share this knowledge and apply it in your organization.

Regards,

Serge


[1] http://www.erlang.org/doc/design_principles/part_frame.html
[2] http://www.erlang.org/pipermail/erlang-questions/2007-August/028392.html


David Mitchell wrote:
> Great tip Serge!
> 
> Like many others, I'm new to Erlang - I've read through the
> "Programming Erlang" book, "Thinking in Erlang" and various PDFs at
> the main Erlang site.  I've written a few trivial bits of code,
> identified and sorted out a bunch of problems I've created, and now
> I'm moving into using Erlang for larger projects.  However, I've never
> come across info about now() and timer.diff() before; like Ahmed, I've
> been using statistics(wall_clock) for profiling purposes.
> 
> Where is that type of info documented?  Is it only in the various
> library APIs (and thus I'll have to work through each of these to get
> a handle on this type of info), or is there some "best practices"-type
> documentation I haven't yet stumbled on?  I don't mind putting in the
> time to do my research, but I suspect I'm not yet across all the
> "good" sources of info.
> 
> Thanks in advance
> 
> Dave M.
> 
> On 13/08/07, Serge Aleynikov <saleyn@REDACTED> wrote:
>> You have several problems with this code:
>>
>> 1. Since you don't provide implementation of generate_lists/3, I assume
>>     it returns a flat list.  In this case in the start function you call:
>>
>>        spawn(?MODULE, run_process,  [Num, Op, Head, Pid]),
>>
>>     if here Head is not a list, whereas run_process/4 expects a list
>>     as the third argument, lists:map/2 function would
>>     crashes the process silently.  You can fix it by changing that line
>>     to:
>>        spawn(?MODULE, run_process,  [Num, Op, [Head], Pid]),
>>
>>
>> 2. Also you use the following call to obtain time:
>>
>>     {_, Wallclock_Time_Since_Last_Call} = statistics(wall_clock).
>>
>>     Wallclock_Time_Since_Last_Call is time in milliseconds, so unless
>>     evaluated function takes more than a millisecond you'd get a 0.
>>
>>     Moreover, unfortunately Wallclock_Time_Since_Last_Call is a *globally
>>     shared* counter, so any process that calls statistics would cause a
>>     reset of this value.  So in a concurrent system where many processes
>>     use this function you'll likely always get a zero.
>>
>>     use T1 = now(), ..., T2 = now(), ... timer:now_diff(T2, T1)
>>     to measure time.
>>
>> Serge
>>
>>
>>
>> Ahmed Ali wrote:
>>> Hi all,
>>>
>>> I've been trying to load test a function in my code. The way I'm doing it is
>>> to generate lists of the data I want to process, start a process for each
>>> list and calculate runtime for each process. The function that I implemented
>>> will call a  WebService operation in a different host for each data.
>>>
>>> I have the code below for this test. what I do is basically run
>>> load_test(10, 1000) to generate 10 lists, each with 1000 data in it.
>>>
>>> The problem is that WebService call is done successfully but I don't get any
>>> output for the statistics. When I run the code sequentially (i.e. instead of
>>> spawn the call to run_process, I call run_process directly) I get the output
>>> with no issues. Is there something that I missed in the code below? I
>>> appreciate your help.
>>>
>>> Best regards,
>>>
>>> Ahmed Al-Issaei
>>>
>>> run_process(Num, Fun, List, _Pid) ->
>>>     statistics(wall_clock),
>>>     io:format("load testing process~n"),
>>>     lists:map(Fun, List),
>>>     {_, Time2} = statistics(wall_clock),
>>>     U2 = Time2 / 1000,
>>>     io:format("Num (~s) done: total time = ~p~n",[Num, U2]).
>>>
>>> start(_N, _Op, [], _Pid) ->
>>>     true;
>>> start(Num, Op, Final, Pid) ->
>>>     [Head | Rest] = Final,
>>>     spawn(?MODULE, run_process,  [Num, Op, Head, Pid]),
>>>     start(Num-1, Op, Rest, Pid).
>>>
>>> load_test(Threads, Size) ->
>>>     List = generate_lists(Threads, Size, []),
>>>     Op = fun(X) -> call_ws_create(X) end,
>>>     start(Threads, Op, List, self()).