[erlang-questions] Issues with stdin on ports

[Per Hedeland]

"Richard A. O'Keefe" <ok@REDACTED> wrote:
>
>On 1/08/2013, at 4:55 AM, Per Hedeland wrote:
>
> [Richard A. O'Keefe wrote:]
>
>>> Per Hedeland wrote: "Erlang open_port/2 *by default* creates
>>> a bi-directional, deadlock-free communication channel to the
>>> external process."  That's still not quite right.  The thing
>>> that governs whether a deadlock is possible or not is the
>>> *protocol* the communicating processes use.  And it is still
>>> the case that an Erlang process may be deadlocked this way;
>>> it's just that the whole Erlang system won't be tied up if it
>>> happens.
>> 
>> No, the statement is quite correct, see above.
>
>No, it really really does depend on the protocol.
>It doesn't matter how the communication is implemented,
>if both processes start by trying to read, that's a
>deadlock.

Wow. I sincerely hope that this astute observation isn't the reason you
have been dragging out this discussion. But I admit it, you are right,
in *any* form of communication between two parties, including human-to-
human, if both parties start out with an unbounded wait for a message
from the other, there will be a deadlock.

With that out of the way, I will try to make amends for my sin, which
amounts to forgetting that you keep wandering away from the subject at
hand. Thus, in the context of a discussion of alleged problems with
bi-directional communication over the channel created by
open_port(spawn), and given your assertion that in case (2) it required
that "both programs have been specially written to communicate in this
way", I incorrectly assumed that you were saying that a protocol
specific to this kind of communication was required to avoid deadlock.

So, to generalize, the channel created by open_port(spawn) doesn't
introduce any risk of deadlock that wasn't already there - but it can't
*remove* a risk of deadlock that exists *without* using this channel.
True, but should be obvious, and is completely irrelevant.

>You appear to be saying that an Erlang process (but not
>the Erlang node containing it) can be blocked forever,
>but somehow that's not a deadlock.  I don't understand.

Yes, of course - a deadlock requires that two parties are both waiting
(perhaps indirectly) for some action from the other. There are obviously
any number of scenarios where an Erlang (or any other) process may block
forever without this constituting a deadlock.

>The issue is that we have two processes connected by
>bounded buffers.

But we don't, since the buffer in the direction from the external
process to the Erlang process is effectively *un*bounded. This is due to
the fact that the VM, making use of non-blocking I/O, will always read
what the external process writes, convert it to an Erlang message, and
place it in the mailbox of the Erlang process that is the controlling
process of the port.

>  And your
>example is a perfect instance of such a protocol, the
>case I labelled (1).

True - as I wrote, it was just something to get you started. And
observing this flaw, you could make the minimal change to make it no
longer fit this case - just repeat the sending line:

2> P ! {self(), {command, "echo " ++ lists:duplicate(100000, $a) ++ "\n"}}, ok.

And lo and behold, it still works without a deadlock, no matter how many
times you repeat that line before doing the receive. (I guess I have to
add that it stops working when the VM runs out of memory...)

>Now let's change your example ever so slightly.
>
>1> P = open_port({spawn, "/bin/cat"}, []).
>2> P ! {self(), {command, lists:duplicate(100000, $a) ++ "\n"}}.
>3> Rec = fun (F, Port, Acc) ->
>             receive {Port, {data, Data}} -> F(F, Port, Acc ++ Data) end
>         end.
>4> Got = Rec(Rec, P, []), ok.
>
>I didn't get as far as step 5, because this deadlocked.

The problem in this example is just that you omitted the 'after 0'
clause of the 'receive'. Thus Rec/3 can never return, and any call of it
will block forever, regardless of what is going on with the
communication. It *is* indeed a deadlock, since both 'cat' and the
Erlang shell are waiting for data from the other - but this is due to
the shell expecting 'cat' to produce an infinite amount of output,
nothing to do with open_port/2.

If you include the 'after 0' clause, the example works just fine:

4> Got = Rec(Rec, P, []), ok.
ok
5> length(Got).
100001

>The big difference here is that cat reads a chunk and echoes
>it back (but P1 isn't interested in reading yet), reads another
>chunk, tries to echo it back, and is blocked because the pipe
>is full.

No, 'cat' is blocked in a read(0, ...) call - easily verified e.g. with
'strace' on Linux.

>Cyclic communications through bounded buffers *do* require
>careful design and it *is* easy to get deadlock that way
>using natural code,

True, but irrelevant.

> and the hard work that went into Erlang's
>port processes is no panacea.

Probably not, but they do not have the problems you think they do. And
in any case, this is irrelevant to the question of whether it would be
useful to add a possibility to close the "out" direction of the channel.
At this point, I have to conclude that you have nothing to contribute
regarding this question.

--Per