[erlang-questions] Re: Shared/Hybrid Heap

Mon Oct 18 11:29:28 CEST 2010

Morten,

With all due respect, your argument is one that I've heard many times
in different settings, and it misses a key point (which Jachym brought up).
My experience from working in development of systems with complex
concurrency patterns is that this argument breaks down completely 
and horribly expensively over time in larger projects.

Saying that there are no fundamental differences between Erlang and C
essentially means that you claim that "correct by design" and "correct by 
convention" are the same thing. By this token, Erlang and Haskell are 
fundamentally the same too, since it really doesn't matter that the Haskell
compiler can verify many more aspects of a program at compile-time than
Erlang can - you can simply avoid making type mistakes in Erlang and 
be done with it!

Your line of reasoning also suggests that we can put great faith in 
micro-benchmarks, since, if you can make it work and make it really 
fast in a small example in C, you can do it equally well in a larger and 
more complex example. 

The place where Erlang and C become very different is when you start
writing code that has "interesting" complexity. Automatic memory 
management may seem like a complete waste, until your program 
becomes sufficiently complex that you can no longer easily keep track
of where you should lock, allocate and free memory. At this point, it is
actually very difficult to outperform a good garbage collector with hand-
written code.

I've seen lots of real-world examples where 
small teams of expert C++ programmers have managed to convince 
management to go with a well-trimmed, elegant execution model in 
C++, only to run in all sorts of problems when lesser programmers were
to start adding features - huge dips in performance and stability, raving
memory leaks, and frantic running around and fire fighting for the 
experts, who are the only ones who seem to manage to stick to the 
conventions and not break stuff. The beginning mantra, "how hard can
it be?", is later answered with "apparently, endlessly hard".

And we haven't even discussed the problem of one of these processes
crashing yet. With two concurrent contexts explicitly writing to the same 
block of memory, ensuring the integrity of that memory block if one of 
the contexts throws an exception will be very, very hard. This is where 
Erlang's "share nothing" philosophy really pays off. The VM may decide
to share data under the hood, but as a programmer, you can be absolutely
certain that the death of another process - even in the same memory space
- cannot ruin the integrity of your own data. It really is as if you had your own
distinct copies of everything.

The problem of crashing processes is a particularly nasty _containment_
problem, but when you scrutinise many of the concurrency models out 
there you soon find that there are usually many issues that are not well
contained. Encapsulation and containment are best handled in the 
programming model, rather than relying on convention, and this is 
exactly the area where there is a fundamental difference.

E.F. Codd, who wrote the twelve (13, really) rules for relational databases, had
this rule as No 12:

Rule 12: The nonsubversion rule:
If the system provides a low-level (record-at-a-time) interface, then that interface 
cannot be used to subvert the system, for example, bypassing a relational security 
or integrity constraint.
(http://en.wikipedia.org/wiki/Codd's_12_rules)

The importance of this rule is often underestimated, and it is quite common to 
ditch it early in favour of performance, since the consequences of subverting
the consistency model are not readily apparent until much later, if even then.

BR,
Ulf W

On 18 Oct 2010, at 09:31, Morten Krogh wrote:

> Hi Jachym
> 
> There is one thing that I probably haven't made clear enough, and it will answer all points basically.
> 
> Erlang has the loop receive loop process that serializes all access to the data.
> The equivalent in C is a single byte. The semantics of communicating with a single byre is well defined in C.
> A C thread communicates with the byte using message passing through the memory bus. The byte only allows get and put.
> Nothing else makes sense.
> 
> The general picture is that you have some units of atomicity and beyond that you need to lock, coordinate access etc.
> 
> Erlang gives you a higher control of the granularity than C, but nothing is fundamentally different. I just object to all the Erlang statements about locks, concurrency etc.
> 
> If you want to get rid of locks and shared memory, you lose concurrency. You can't have both. You can serialize everything. In C, that would be one thread. In Erlang,
> one process. If you want concurrency, you need smaller blocks of data, which then needs to be coordinated.
> 
> You can imagine an axis
> 
> concurrency ~ locks, deadlocks etc  ~ high performance   in one end.
> 
> serialized access ~ no locks ~ single threaded  ~ simple programing model    in the other end.
> 
> 
> C actually beats Erlang in both ends of the axis (not the programming model, of course, Erlang syntax is always simpler).
> 
> Erlang's sweet spot is in the middle. It is so easy to package an amount of data into a serialized agent with controller logic.
> 
> The most concurrent program is not an erlang program, that would be a C program (or even lower) with a loop on each core, where all iterations of all loops updated the memory in all kinds of ways. Each iteration of each loop is a "context switch".
> 
> Erlang is of course very convenient. You get a whole "block of bytes" with serialized access, and a programming language to control the "gate" to the block of bytes.
> 
> The hardware manufacturers could make something like Erlang, if they wanted to. Chop the memory into blocks, and give each block serialized access and a small programmable controller sitting at the gate of that block. They would lose performance and concurrency if they did.
> 
> It is all the same. The only question is at what granularity you serialize your data.
> 
> Jachym, I will not go through your points. You can imagine my response. It is basically :
> 
> One byte in C is the equivalent of one Erlang process. One byte doesn't need any controller logic besides get and put, and a guaranteed atomicity of get and put.
> 
> Cheers,
> 
> Morten
> 
> 
> 
> On 10/17/10 11:46 PM, Jachym Holecek wrote:
>> # Morten Krogh 2010-10-17:
>>> The problem is of course that a process like this
>>> 
>>> loop(State) ->
>>>     receive
>>>     {From, get, X} ->
>>>          From ! {result, get(X, State)},
>>>         loop(State)
>>> 
>>>     etc.....
>>> 
>>> Behaves exactly like shared memory seen from other processes point of view.
>>> 
>>> Writing
>>> 
>>> Pid ! {self(), get, 27}
>>> receive
>>>     {result, Result} ->
>>>         Result
>>> end.
>>> 
>>> is like writing
>>> 
>>> get(X, Pid) in C (the language) where Pid is pointer to a shared data
>>> structure in C.
>> "Apples and oranges" as they say. In your Erlang example:
>> 
>>   * Server explicitely agrees to provide some of its data.
>>   * Server is free to change this decision.
>>   * Client has to follow certain protocol to access this data.
>>   * All accesses to "shared" data are serialized.
>>   * It is "physically impossible" to obtain inconsistent snaphost of the data.
>>   * It is "physically impossible" to bypass server's control.
>>   * This is guaranteed at language level.
>>   * Your example is implemented completely within this well-defined semantics.
>> 
>> Contrast that with C:
>> 
>>   * All threads have equal access right to whole process address space.
>>   * Sychronization can be done, but is a mere convention (spinlocks/mutexes).
>>   * Data consistency can be done, but is a mere convetion (memory barriers).
>>   * At language level memory access semantics is more or less undefined.
>>   * Mutexes/memory barriers have to be done in assembly, ie. outside C.
>> 
>> A little more fun on the C side:
>> 
>>   * Compiler may reorder loads/stores (somewhat) unless you're careful.
>>   * CPU may reorder loads/stores (somewhat) unless you're careful.
>>   * Cache may do whatever it wants to (somewhat) unless you're careful.
>> 
>> Normally these thing will be taken care of by whatever threads implementation
>> you use (in architecture- and compiler-specific way), but if we're talking at
>> language level then there's no way (as far as I know) to achieve sane shared
>> memory semantics in C alone (if we admit the existence of concurrent threads
>> of execution which the language knows nothing about, of course).
>> 
>> But sure -- you can _model_ Erlang-like processes in C, and you can _model_
>> C-like shared memory in Erlang (including all of the quirks above if you
>> really wanted to). But at that point you're imposing correspondence of some
>> sort by brute force; as opposed to capturing inherent properties.
>> 
>> Just my two pence...
>> 
>> Regards,
>> 	-- Jachym
>> 
>> ________________________________________________________________
>> erlang-questions (at) erlang.org mailing list.
>> See http://www.erlang.org/faq.html
>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
>> 
> 
> 
> ________________________________________________________________
> erlang-questions (at) erlang.org mailing list.
> See http://www.erlang.org/faq.html
> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
> 

Ulf Wiger, CTO, Erlang Solutions, Ltd.
http://erlang-solutions.com