Hi,<br><br> For me it look as messages comes from port really fast. They coming fast because eof couldn't be detected and most of the messages are {data, []}. It is because <strong><a name="open_port/2"><strong><code>
</code></strong></a></strong><a name="open_port/2"><strong></strong><strong><strong><a name="open_port/2"><strong><code>PortSettings </code></strong></a></strong></strong><a name="open_port/2">in </a><strong><strong><a name="open_port/2">
<strong><code></code></strong></a></strong></strong></a><strong><a name="open_port/2"><strong><code>open_port(PortName, PortSettings) </code></strong></a></strong><a name="open_port/2">have to contain </a>something more than
<strong><a name="open_port/2"><strong><strong></strong></strong></a><strong><strong><a name="open_port/2"><strong><code>[eof]</code></strong></a></strong></strong></strong>. Like for example <strong><a name="open_port/2">
<strong><strong></strong></strong></a><strong><strong><a name="open_port/2"><strong><code>[eof, {line, 1000}]</code></strong></a></strong></strong></strong>. Then eof is detected and no more data is delivered by port.<br>
<br> The above is based on my observations. I am not an expert in erlang field so please forgive my ignorance in case if I missed something obvious.<br><br>-- <br>Kind Regards, <br>Daniel Kwiecinski<br><strong><a name="open_port/2">
<strong></strong></a></strong><br><br><div><span class="gmail_quote">2007/6/29, Fredrik Svahn <<a href="mailto:fredrik.svahn@gmail.com">fredrik.svahn@gmail.com</a>>:</span><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Sorry for double posting, it seems I have misconfigured something at trapexit... <br><br>Fredrik Svahn wrote:<br>I have also been frustrated by the way the io operations work when
attempting to speed up a few of the example programs for the language
shootout. The reverse-complement program for instance (which is approx.
60 times slower than the corresponding c program) spends 80% of its
time reading from stdio, and I assume writing out the results are quite
costly too.
<br>
<br>Just for fun I made a small patch to efile to allow for
stdin/stdout to be opened as files. It probably has a lot of nasty side
effects which I cannot even imagine in the worst of my nightmares, but
the results for reading and writing are stunning. The "file" approach
clocks in at 0.26 seconds for reading a large file from stdin and
writing it to stdout. Corresponding results for a port program is 1.2
seconds with the normal io approach measuring in at 16.9 seconds. <br>
<br>
I guess reading from stdin is not much of a problem for most Erlang
applications which are supposed to be robust scalable systems staying
up for years. I also think that this has been discussed before,
probably at great length, although I cannot find any relevant posts at
the moment. But now with escript maybe it might be a bit more
interesting to have <span id="st" name="st" class="st">fast</span> io operations for stdin/stdout, at least for
unix systems? <br>
<br>
I haven't looked at memory consumption, yet, but I expect the result
should be the same as for Ulf, i.e. port programs build up large heaps
if they cannot handle the messages really really really <span id="st" name="st" class="st">fast</span>, while the
file and normal io approach should not really consume much more memory
than the buffer.
<br>
<br>
BR /Fredrik
<br>
<br>
Test-program:
<br><div><div><br>
-module(io_test).
<br>
-export([file/0,port/0,normal/0,fileio/0,portio/0,normalio/0]).
<br>
-define(bufsize, 2048).
<br>
<br>
file()-> io:format("~n~p~n",[timer:tc(?MODULE, fileio, [])]), halt().
<br>
port()-> io:format("~n~p~n",[timer:tc(?MODULE, portio, [])]), halt().
<br>
normal()-> io:format("~n~p~n",[timer:tc(?MODULE, normalio, [])]), halt().
<br>
<br>
fileio()->
<br>
{ok,StdIn}=file:open("<stdin>",[raw, binary, read]),
<br>
{ok,StdOut}=file:open("<stdout>",[raw, binary, write]),
<br>
fileio(StdIn, StdOut).
<br>
<br>
fileio(StdIn, StdOut) ->
<br>
case file:read(StdIn,?bufsize) of
<br>
<span id="__firefox-findbar-search-id" style="padding: 0pt; background-color: yellow; color: black; display: inline; font-size: inherit;">eof</span> -> ok;
<br>
{ok, Data} ->
<br>
file:write(StdOut, Data),
<br>
fileio(StdIn, StdOut)
<br>
end.
<br>
<br>
portio()->
<br>
Port=open_port({fd, 0, 1},[<span id="__firefox-findbar-search-id" style="padding: 0pt; background-color: yellow; color: black; display: inline; font-size: inherit;">eof</span>]),
<br>
portio(Port),
<br>
port_close(Port).
<br>
<br>
portio(Port)->
<br>
receive
<br>
{Port, {data, Data}} ->
<br>
port_command(Port, Data),
<br>
portio(Port);
<br>
{_Port, <span id="__firefox-findbar-search-id" style="padding: 0pt; background-color: yellow; color: black; display: inline; font-size: inherit;">eof</span>} -> ok
<br>
end.
<br>
<br>
normalio() ->
<br>
case io:get_chars('',?bufsize) of
<br>
<span id="__firefox-findbar-search-id" style="padding: 0pt; background-color: yellow; color: black; display: inline; font-size: inherit;">eof</span> -> ok;
<br>
Data ->
<br>
io:put_chars(Data),
<br>
normalio()
<br>
end.
<br>
<br>
</div></div>
<br>
<br>
Command-lines:
<br><div><div><br>
$ erl -noinput -run io_test file < txt > tmp-file ; tail -n 1 tmp-file
<br>
{259951,ok}
<br>
$ erl -noinput -run io_test port < txt > tmp-port ; tail -n 1 tmp-port
<br>
{1193521,true}
<br>
$ erl -noinput -noshell -run io_test normal < txt > tmp-normal ; tail -n 1 tmp-normal
<br>
{16946068,ok}
<br>
</div></div>
<br>
<br>
Patch for unix on R11B-5:
<br>
diff ./erts/emulator/drivers/unix/unix_efile.c ./erts/emulator/drivers/unix/unix_efile.c.old
<br>
781,789c781
<br>
<
<br>
< if (strcmp(name, "<stdin>") == 0) {
<br>
< fd = 0;
<br>
< } else if (strcmp(name, "<stdout>") == 0) {
<br>
< fd = 1;
<br>
< } else {
<br>
< fd = open(name, mode, FILE_MODE);
<br>
< }
<br>
<
<br>
---
<br>
> fd = open(name, mode, FILE_MODE);<br><br><br><br><br><div><div><span class="e" id="q_1137790de7bebbec_1"><span class="gmail_quote">On 6/26/07, <b class="gmail_sendername">Ulf Wiger (TN/EAB)</b> <<a href="mailto:ulf.wiger@ericsson.com" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
ulf.wiger@ericsson.com
</a>> wrote:</span></span></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div><div><span class="e" id="q_1137790de7bebbec_3">
<br>
<p><font face="Arial" size="2">I submitted a sum-file entry to the shootout, which worked</font>
<br><font face="Arial" size="2">nicely in my environment(*), but failed miserably in the </font>
<br><font face="Arial" size="2">official benchmark.</font>
</p>
<p><a href="http://shootout.alioth.debian.org/gp4/benchmark.php?test=sumcol&lang=hipe&id=2" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)"><u><font color="#0000ff" face="Arial" size="2">http://shootout.alioth.debian.org/gp4/benchmark.php?test=sumcol&lang=hipe&id=2
</font></u></a>
</p>
<br>
<p><font face="Arial" size="2">It uses the (admittedly undocumented) command-line flag for</font>
<br><font face="Arial" size="2">installing a custom user process, and opens stdin in line-</font>
<br><font face="Arial" size="2">oriented mode.</font>
</p>
<p><font face="Arial" size="2">The problem is that it runs out of memory. As far as I can make</font>
<br><font face="Arial" size="2">out, it's because the emulator chops up lines and sends them</font>
<br><font face="Arial" size="2">to the process at such a high rate that, even though the </font>
<br><font face="Arial" size="2">process is in a tight loop and doing minimal work on each item,</font>
<br><font face="Arial" size="2">it can't stop the message queue from building up.</font>
</p>
<p><font face="Arial" size="2">This has disastrous effects when the input file is large enough.</font>
</p>
<p><font face="Arial" size="2">I realise that the feature is undocumented, but perhaps it's still</font>
<br><font face="Arial" size="2">a valid point - some sort of generic flow-control on ports, </font>
<br><font face="Arial" size="2">similar to the {active, bool()} on sockets, would be just the</font>
<br><font face="Arial" size="2">thing here.</font>
</p>
<p><font face="Arial" size="2">(*) I realise that I tested it in an NFS-mounted disk (on a clearcase-</font>
<br><font face="Arial" size="2">enabled file system at that). This might have given the port </font>
<br><font face="Arial" size="2">sufficient flow control that the program lasted a bit longer, at least.</font>
</p></span></div>
</div>
<br>_______________________________________________<br>erlang-questions mailing list<br><a href="mailto:erlang-questions@erlang.org" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">erlang-questions@erlang.org
</a><br>
<a href="http://www.erlang.org/mailman/listinfo/erlang-questions" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">http://www.erlang.org/mailman/listinfo/erlang-questions</a><br></blockquote></div><br>
<br>_______________________________________________<br>erlang-questions mailing list<br><a onclick="return top.js.OpenExtLink(window,event,this)" href="mailto:erlang-questions@erlang.org">erlang-questions@erlang.org</a><br>
<a onclick="return top.js.OpenExtLink(window,event,this)" href="http://www.erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://www.erlang.org/mailman/listinfo/erlang-questions</a><br></blockquote></div><br>
<br clear="all"><br>