<div dir="ltr">Thanks for that explanation, Lukas, that definitely makes sense. I've gone with generating a simple shell script that invokes tar and pipes it to split.<div><br></div><div>n</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Jul 17, 2016 at 12:44 AM, Lukas Larsson <span dir="ltr"><<a href="mailto:garazdawi@gmail.com" target="_blank">garazdawi@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hello,<div><br></div><div>There is no back-pressure mechanism for open_port, so when you call it on "cat /dev/random" you will get an endless stream of data into the process' message queue which Erlang most likely cannot keep up with and that makes the memory grow until it explodes. In order to get back-pressure you have to put something in between "cat /dev/random" and the erlang port which you processes can talk to and request more bytes instead of it just sending as much as it can. A small bash script which sends X bytes for every Y bytes it receives could do the trick.</div><div><br></div><div>Lukas<br><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="h5">On Sat, Jul 16, 2016 at 7:13 PM, Nathan Fiedler <span dir="ltr"><<a href="mailto:nathanfiedler@gmail.com" target="_blank">nathanfiedler@gmail.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5"><div dir="ltr"><font face="monospace, monospace">My ultimate goal is to run two external processes, with the output of one feeding into the input of the other, a la shell piping. So something like `zfs send | zfs recv` or `tar | split`, but from within my Erlang application. Below is my simple example that fairly quickly balloons memory out of control and eventually explodes (i.e. the OS kills it).<br><br>#!/usr/bin/env escript<br>%%!<br><br>main(_Args) -><br> SendCmd = "cat /dev/random",<br> %<br> % not using 'binary' with open_port makes memory size grow really fast,</font><div><font face="monospace, monospace"> % but that's fine, I want that option anyway, just pointing it out<br> %<br> SendPort = erlang:open_port({spawn, SendCmd}, [exit_status, binary]),<br> RecvCmd = "strings",<br> RecvPort = erlang:open_port({spawn, RecvCmd}, [exit_status, binary]),<br> {ok, 0} = pipe_until_exit(SendPort, RecvPort, 0),<br> ensure_port_closed(SendPort),<br> ensure_port_closed(RecvPort),<br> ok.<br><br>pipe_until_exit(SendPort, RecvPort, N) -><br> %<br> % invoking garbage_collect/0 helps a little bit, but memory still grows<br> %<br> io:format("iteration ~w~n", [N]),<br> receive<br> {SendPort, {exit_status, Status}} -><br> {ok, Status};<br> {RecvPort, {exit_status, Status}} -><br> io:format("error: receive port exited ~w~n", [Status]);<br> {SendPort, {data, Data}} -><br> true = erlang:port_command(RecvPort, Data),<br> pipe_until_exit(SendPort, RecvPort, N + 1);<br> {RecvPort, {data, _Data}} -><br> pipe_until_exit(SendPort, RecvPort, N + 1);<br> Msg -><br> io:format("some other message: ~w", [Msg])<br> end.<br><br>ensure_port_closed(Port) -><br> case erlang:port_info(Port) of<br> undefined -> ok;<br> _ -> erlang:port_close(Port)<br> end.<br></font><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">My assumption is that I am doing something wrong, but I can't see it. I'll explore other solutions, such as simply generating a shell script and invoking it. In the mean time, if you can see something obvious, please let me know.</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">Yes, I am aware of os:cmd/1, but that does not return the exit code, which I really want to have so I know that something at least appeared to work (zfs send|zfs recv and tar|split do not generate any success indication other than exit code).</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">Thanks</font></div></div><span><font color="#888888"><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">n</font></div><div><br></div></font></span></div>
<br></div></div>_______________________________________________<br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a><br>
<a href="http://erlang.org/mailman/listinfo/erlang-questions" rel="noreferrer" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
<br></blockquote></div><br></div></div></div>
</blockquote></div><br></div>