[erlang-questions] Issues with stdin on ports

[Anthony Grimes]

Once again, I'm sorry I haven't got the terminology right.

Sure, that's an option. I'm still unclear why I can do this in every
other language I use but not Erlang. I find it hard to believe that all
other languages with this capability are doing something utterly stupid,
but Erlang, being the best language in the known universe, has gotten it
utterly correct. But once again, I'll concede.

Thanks for the responses and teaching me a thing or two, Mr. O'Keefe!

> Richard A. O'Keefe <mailto:ok@REDACTED>
> July 29, 2013 11:23 PM
> On 30/07/2013, at 6:02 PM, Anthony Grimes wrote:
>> I'm simply asking how I'm supposed to deal with programs that I have *no* control over that wait for a EOF on their input to start producing data. I just want to send a ^D.
>
> End of file and Control-D are two different things.
> UNIX technically doesn't have a character that signifies end of file.
> There is an "EOF" character, but what it does is "send the current line NOW",
> and sending a line with *no* characters, not even a newline, causes a read()
> to return 0, and _that_ is the end of file indication.
> Since 1979, when I started using UNIX, my end-of-file character has been
> Control-Z, to be compatible with the other computer systems I was using.
> I am rather fed up with readline-infested programs that insist on Control-D
> instead of accepting the EOF character I set up.
>
> And a program that waits for all of its input before producing any data
> is by definition not a filter.
>
> There is a very very simple technique.
> (1) Create a temporary file.
> (2) Create a pipe, telling that pipe to write to the temporary file.
> (3) Send your data to the pipe and close the pipe.
> (4) Now read the temporary file.
>
> I still don't understand why you can't do this/
>
>> I have examples of programs that do that and a thousand ways to do it in other languages, and I'm being told it is disastrous to do. If the solution is to just keep doing what everybody has been doing in Erlang for years which is write giant hack middleman programs to do it,
>
> No, that is NOT what everybody has been doing in Erlang.
> Creating a temporary file is simple and safe and easy to program.
>
> Anthony Grimes <mailto:i@REDACTED>
> July 29, 2013 1:20 AM
> Yeah, re-reading your post a couple of times, I think we might be on
> the wrong page or something. Here is a low level example of what I'd
> like to be able to do in Erlang. This is a Clojure repl session where
> I interact with the 'cat' program via the built in Java Process library:
>
> user=> (def proc (.exec (Runtime/getRuntime) (into-array String ["cat"])))
> #'user/proc
> user=> (def stdin (.getOutputStream proc))
> #'user/stdin
> user=> (def stdout (.getInputStream proc))
> #'user/stdout
> user=> (.write stdin (.getBytes "Hi!"))
> nil
> user=> (.close stdin)
> nil
> user=> (let [arr (byte-array 3)] (.read stdout arr) (String. arr))
> "Hi!"
>
> Lots of unix programs work like this. We have cat in this example, but
> grep, wc, and various others work like that as well. It is this easy
> or easier to do the same thing in every other language I can think of.
> If it's fundamentally a bad thing, I'm surprised these programs work
> like that in the first place and that these languages support this. It
> seems to be an entirely common place, basic feature any remotely high
> level programming language.
>
> Perhaps this example and clarification will clear things up!
>
> -Anthony
> Richard A. O'Keefe <mailto:ok@REDACTED>
> July 29, 2013 12:15 AM
> On 29/07/2013, at 4:14 PM, Anthony Grimes wrote:
>
>> and communicating with external processes in Erlang. They seem to have
>> at least one particular fatal flaw which prevents them from being very
>> useful to me, and that is that there is no way to close stdin (and send
>> EOF) and then also read from the process's stdout. For example, I cannot
>> use a port to start the 'cat' program which listens on stdin for data
>> and waits for EOF and then echos that data back to you. I can do the
>> first part, which is send it data on stdin, but the only way for me to
>> close it is to call port_close and close the entire process.
>
> Note that "only send data to a command" and "only receive data from a
> command" are the traditional ways for a UNIX program to communicate
> with another over a pipe.  popen(<command>, "r") reads the output of
> the command and popen(<command>, "w") writes to the input of the command.
> There isn't even any standard _term_ for talking about connecting to both
> stdin and stdout of a command in UNIX, and that's because it's an
> incredibly easy way to deadlock.
>
>> This issue prevents Erlang users from doing any even slightly more than
>> trivial communication with external processes without having some kind
>> of middleman program that handles the creation of the actual process you
>> need to talk to and looks for a specific byte sequence to indicate 'EOF'.
>
> Just like it prevents C users from doing the same thing.
> Unless they fake something up using named pipes or UNIX-domain sockets.
> (Or message queues.  I do wish Mac OS X implemented rather more of POSIX...)
>
> Unix anonymous pipes are simply the wrong tool for the job in _any_
> programming language.
>
> The historic way to do "slightly more than trivial communication with
> external processes" has been to set the external processes up as C nodes
> or to use sockets.
>
>
>
>
> Anthony Grimes <mailto:i@REDACTED>
> July 28, 2013 9:14 PM
> Howdy folks.
>
> I unfortunately have not been able to use Erlang for most of what I've
> been doing lately because of a long standing issue with Erlang ports
> that I'd like to start a discussion about here.
>
> As far as I am aware, ports are generally the only option for creating
> and communicating with external processes in Erlang. They seem to have
> at least one particular fatal flaw which prevents them from being very
> useful to me, and that is that there is no way to close stdin (and send
> EOF) and then also read from the process's stdout. For example, I cannot
> use a port to start the 'cat' program which listens on stdin for data
> and waits for EOF and then echos that data back to you. I can do the
> first part, which is send it data on stdin, but the only way for me to
> close it is to call port_close and close the entire process.
>
> This issue prevents Erlang users from doing any even slightly more than
> trivial communication with external processes without having some kind
> of middleman program that handles the creation of the actual process you
> need to talk to and looks for a specific byte sequence to indicate 'EOF'.
>
> I could totally be wrong, but it seems like we need something other than
> just port_close. Something like
> http://www.erlang.org/doc/man/gen_tcp.html#shutdown-2
> <http://www.erlang.org/doc/man/gen_tcp.html#shutdown-2> which lets you say
> "Hey, I want to close the stdin of this process but still read from its
> stdout." or something similar. I could be totally off track on what a
> good solution would be.
>
> So I'm wondering if people are aware of this problem, and I'd like to
> make sure that people think it is an actual problem that should be
> fixed. I'm also curious what people think a good solution to the problem
> would be. I'm not sure I have the time/particular skill set to fix it
> given that the port code is some pretty obscure (to me) C code, but
> starting conversation seems like a good way to begin.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20130729/7a40549a/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: compose-unknown-contact.jpg
Type: image/jpeg
Size: 770 bytes
Desc: not available
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20130729/7a40549a/attachment.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: postbox-contact.jpg
Type: image/jpeg
Size: 1188 bytes
Desc: not available
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20130729/7a40549a/attachment-0001.jpg>