It seems that block counting is not necessary. I'm taking same performance (~5MB/s measured at 10 million messages) even this version:<br><br>-module(testLogger).<br><br>-compile(inline).<br><br>-export([listener/1, start/1, write/2]).<br>
<br>start(Filename) -><br> Pid = spawn(?MODULE, listener, [Filename]),<br> register(?MODULE, Pid).<br><br>listener(Filename) -><br> case file:open(Filename, [append, raw]) of<br> {ok, Fd} -> listener(Fd, 0, []);<br>
{error, Reason} -> error_logger:error_msg(Reason)<br> end.<br><br>listener(Fd, N, [] = SoFar) -><br> receive<br> {data, Data} -> listener(Fd, N + 1, [Data]);<br> {count, From} -><br> From ! {count, N}, listener(Fd, N, SoFar);<br>
stop -> ok;<br> Msg -><br> io:format("Unexpected message: ~p~n", [Msg]),<br> listener(Fd, N, SoFar)<br> end;<br>listener(Fd, N, SoFar) -><br> receive<br> {data, Data} -> listener(Fd, N + 1, [Data | SoFar]);<br>
{count, From} -><br> From ! {count, N}, listener(Fd, N, SoFar);<br> stop -> ok;<br> Msg -><br> io:format("Unexpected message: ~p~n", [Msg]),<br> listener(Fd, N, SoFar)<br>
after 0 -> flush(Fd, N, SoFar)<br> end.<br><br>flush(Fd, N, Data) -><br> file:write(Fd, lists:reverse(Data)),<br> listener(Fd, N, []).<br><br>write(_Data, 0) -> ok;<br>write(Data, N) -><br> (?MODULE) ! {data, Data}, write(Data, N - 1).<br>
<br><div class="gmail_quote">On Fri, Apr 24, 2009 at 8:32 PM, Hynek Vychodil <span dir="ltr"><<a href="mailto:vychodil.hynek@gmail.com">vychodil.hynek@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Try this one, I guess you will be surprised ;-)<br><br>-define(BS, 65536).<br><br>start(Filename) -><br> Pid = spawn(?MODULE, listener, [Filename]),<br> register(?MODULE, Pid).<br><br>listener(Filename) -><br>
case file:open(Filename, [append, raw]) of<br>
{ok, Fd} -> listener(Fd, 0, [], 0);<div class="im"><br> {error, Reason} -> error_logger:error_msg(Reason)<br> end.<br><br></div>listener(Fd, N, [] = SoFar, 0) -><br> receive<br> {data, Data} -><br>
listener(Fd, N + 1, [Data], iolist_size(Data));<br>
{count, From} -><br> From ! {count, N}, listener(Fd, N, SoFar, 0);<br> stop -> ok;<div class="im"><br> Msg -><br> io:format("Unexpected message: ~p~n", [Msg]),<br></div>
listener(Fd, N, SoFar, 0)<br>
end;<br>listener(Fd, N, SoFar, S) when S > (?BS) -><br> file:write(Fd, lists:reverse(SoFar)),<br> listener(Fd, N, [], 0);<br>listener(Fd, N, SoFar, S) -><br> receive<br> {data, Data} -><br> listener(Fd, N + 1, [Data | SoFar],<br>
S + iolist_size(Data));<br> {count, From} -><br> From ! {count, N}, listener(Fd, N, SoFar, S);<br> stop -> ok;<div class="im"><br> Msg -><br> io:format("Unexpected message: ~p~n", [Msg]),<br>
</div>
listener(Fd, N, SoFar, S)<br> after 0 -><br> file:write(Fd, lists:reverse(SoFar)),<br> listener(Fd, N, [], 0)<br> end.<br><br>write(_Data, 0) -> ok;<br>write(Data, N) -><br>
(?MODULE) ! {data, Data}, write(Data, N - 1).<br><br><div class="gmail_quote"><div class="im">On Fri, Apr 24, 2009 at 4:23 PM, chaitanya Chalasani <span dir="ltr"><<a href="mailto:chaitanya.chalasani@gmail.com" target="_blank">chaitanya.chalasani@gmail.com</a>></span> wrote:<br>
</div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div><div></div><div class="h5">
<div>
Hi,<br>
<br>
I am facing performance issue with my file logger. The file logger is an event handler holding the file reference is its state. The performance of the module is way ahead in the smp disabled erl shell compared with default smp enabled erl shell.<br>
<br>
I tried ...<br>
case#1<br>
<i>Erlang R13B (erts-5.7.1) [source] [rq:1] [async-threads:0] [hipe] [kernel-poll:false]</i><br>
<br>
<i>Eshell V5.7.1 (abort with ^G)</i><br>
<i>1> testLogger:start("test.log").</i><br>
<i>true</i><br>
<i>2> testLogger:write("Testing...",1000000).</i><br>
<i>{ok,done}</i><br>
<i>3></i><br>
<br>
and <br>
<br>
case#2<br>
<i>Erlang R13B (erts-5.7.1) [source] [smp:2:2] [rq:2] [async-threads:0] [hipe] [kernel-poll:false]</i><br>
<br>
<i>Eshell V5.7.1 (abort with ^G)</i><br>
<i>1> testLogger:start("test.log").</i><br>
<i>true</i><br>
<i>2> testLogger:write("Testing...",1000000).</i><br>
<i>{ok,done}</i><br>
<i>3></i><br>
<br>
<br>
What I observed was -<br>
Case 1: <br>
1. The testLogger:write("Testing...",1000000) did not return back the control to the shell until all the messages are written to log.<br>
2. The data was written at 400 KB/s<br>
<br>
Case 2:<br>
1. The testLogger:write("Testing...",1000000) return back the control to the shell with {ok, done} after a short while but the messages where being written to log even after that.<br>
2. The data was written at 4 KB/s<br>
3. I checked in the pman tool it shows current funtion as io:wait_io_mon_reply/2 and the messages in the message pool are about 880k.<br>
<br>
Below is the code which I tested.<br>
<br>
<i>start(Filename) -> register(testLogger,spawn(testLogger,listener,[Filename])).</i><br>
<br>
<i>listener(Filename) -></i><br>
<i> case file:open(Filename,[append]) of</i><br>
<i> {ok,Fd} -> listener(Fd,0);</i><br>
<i> {error,Reason} -> error_logger:error_msg(Reason)</i><br>
<i> end.</i><br>
<br>
<i>listener(Fd,N) -></i><br>
<i> receive</i><br>
<i> {data,Data} -> file:write(Fd,Data), listener(Fd,N+1);</i><br>
<i> {count,From} -> From ! {count,N}, listener(Fd,N);</i><br>
<i> {stop,From} -> From ! {stop,success}</i><br>
<i> end.</i><br>
<br>
<i>write(_Data,0) -> {ok,done};</i><br>
<i>write(Data,N) -> testLogger ! {data,Data}, write(Data,N-1).</i><br>
<br>
Is there a problem with my code or is it a known issue?<br>
<br>
<table width="100%" cellpadding="0" cellspacing="0">
<tbody><tr>
<td>
<b><font size="2"><font color="#808080">------</font></font></b><br><font color="#888888">
<b><font size="2"><font color="#808080">CHAITANYA CHALASANI</font></font></b>
</font></td>
</tr>
</tbody></table>
</div>
<br></div></div><div class="im">_______________________________________________<br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a><br>
<a href="http://www.erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://www.erlang.org/mailman/listinfo/erlang-questions</a><br></div></blockquote></div><br><br clear="all"><div><div></div><div class="h5">
<br>-- <br>--Hynek (Pichi) Vychodil<br>
<br>Analyze your data in minutes. Share your insights instantly. Thrill your boss. Be a data hero!<br>Try Good Data now for free: <a href="http://www.gooddata.com" target="_blank">www.gooddata.com</a><br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br>--Hynek (Pichi) Vychodil<br><br>Analyze your data in minutes. Share your insights instantly. Thrill your boss. Be a data hero!<br>Try Good Data now for free: <a href="http://www.gooddata.com">www.gooddata.com</a><br>