[erlang-questions] os:cmd performance

[Michał Ptaszek]

Hey,

I am trying to run some shell commands from Erlang, using os:cmd/1:

  os:cmd("cat /tmp/huge_file | grep 'some_regexp' | sort -u >
/tmp/output_file").

Where huge_file is ~4GB big, the output_file is expected to occupy ~150MB
on disk. When running this command from Erlang it takes ~45-50 minutes to
create the output file, however when ran directly from the shell (bash)
everything happens within 10 minutes.
It's not a matter of FS cache (I "echo 3 > /proc/sys/vm/drop_caches" right
before each test), or external load (it's a physical machine, completely
idle during the test). Erlang version: R16B.

My question here is not "how to do it?", but rather "why os:cmd/1 is so
inefficient?", even though the output returned to the caller is empty
(stdout is redirected to the disk file, there are no stderr printouts). Or
maybe I'm doing something entirely stupid that should be avoided...

Any clues/hints?
Thanks,
Michal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20130729/a3c4e08e/attachment.htm>