<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 TRANSITIONAL//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; CHARSET=UTF-8">
<META NAME="GENERATOR" CONTENT="GtkHTML/3.26.0">
</HEAD>
<BODY>
Thank you very much it works great. <BR>
<BR>
We are using gen_event with multiple file logging handlers for each module and also simultaneous logging into different files for debug, normal and panic messages. In this case we have to use selective receive. <BR>
<BR>
I think this suffices our logging hunger of about 1000 transactions/sec. I need to test more for further scaling.<BR>
<BR>
On Fri, 2009-04-24 at 22:19 +0200, Hynek Vychodil wrote:<BR>
<BLOCKQUOTE TYPE=CITE>
It seems that block counting is not necessary. I'm taking same performance (~5MB/s measured at 10 million messages) even this version:<BR>
<BR>
-module(testLogger).<BR>
<BR>
-compile(inline).<BR>
<BR>
-export([listener/1, start/1, write/2]).<BR>
<BR>
start(Filename) -><BR>
Pid = spawn(?MODULE, listener, [Filename]),<BR>
register(?MODULE, Pid).<BR>
<BR>
listener(Filename) -><BR>
case file:open(Filename, [append, raw]) of<BR>
{ok, Fd} -> listener(Fd, 0, []);<BR>
{error, Reason} -> error_logger:error_msg(Reason)<BR>
end.<BR>
<BR>
listener(Fd, N, [] = SoFar) -><BR>
receive<BR>
{data, Data} -> listener(Fd, N + 1, [Data]);<BR>
{count, From} -><BR>
From ! {count, N}, listener(Fd, N, SoFar);<BR>
stop -> ok;<BR>
Msg -><BR>
io:format("Unexpected message: ~p~n", [Msg]),<BR>
listener(Fd, N, SoFar)<BR>
end;<BR>
listener(Fd, N, SoFar) -><BR>
receive<BR>
{data, Data} -> listener(Fd, N + 1, [Data | SoFar]);<BR>
{count, From} -><BR>
From ! {count, N}, listener(Fd, N, SoFar);<BR>
stop -> ok;<BR>
Msg -><BR>
io:format("Unexpected message: ~p~n", [Msg]),<BR>
listener(Fd, N, SoFar)<BR>
after 0 -> flush(Fd, N, SoFar)<BR>
end.<BR>
<BR>
flush(Fd, N, Data) -><BR>
file:write(Fd, lists:reverse(Data)),<BR>
listener(Fd, N, []).<BR>
<BR>
write(_Data, 0) -> ok;<BR>
write(Data, N) -><BR>
(?MODULE) ! {data, Data}, write(Data, N - 1).<BR>
<BR>
</BLOCKQUOTE>
<BLOCKQUOTE TYPE=CITE>
On Fri, Apr 24, 2009 at 8:32 PM, Hynek Vychodil <<A HREF="mailto:vychodil.hynek@gmail.com">vychodil.hynek@gmail.com</A>> wrote:<BR>
<BLOCKQUOTE>
Try this one, I guess you will be surprised ;-)<BR>
<BR>
-define(BS, 65536).<BR>
<BR>
start(Filename) -><BR>
Pid = spawn(?MODULE, listener, [Filename]),<BR>
register(?MODULE, Pid).<BR>
<BR>
listener(Filename) -><BR>
case file:open(Filename, [append, raw]) of<BR>
{ok, Fd} -> listener(Fd, 0, [], 0);
</BLOCKQUOTE>
</BLOCKQUOTE>
<BLOCKQUOTE TYPE=CITE>
<BLOCKQUOTE>
<BR>
{error, Reason} -> error_logger:error_msg(Reason)<BR>
end.<BR>
<BR>
<BR>
</BLOCKQUOTE>
</BLOCKQUOTE>
<BLOCKQUOTE TYPE=CITE>
<BLOCKQUOTE>
listener(Fd, N, [] = SoFar, 0) -><BR>
receive<BR>
{data, Data} -><BR>
listener(Fd, N + 1, [Data], iolist_size(Data));<BR>
{count, From} -><BR>
From ! {count, N}, listener(Fd, N, SoFar, 0);<BR>
stop -> ok;
</BLOCKQUOTE>
</BLOCKQUOTE>
<BLOCKQUOTE TYPE=CITE>
<BLOCKQUOTE>
<BR>
Msg -><BR>
io:format("Unexpected message: ~p~n", [Msg]),<BR>
<BR>
</BLOCKQUOTE>
</BLOCKQUOTE>
<BLOCKQUOTE TYPE=CITE>
<BLOCKQUOTE>
listener(Fd, N, SoFar, 0)<BR>
end;<BR>
listener(Fd, N, SoFar, S) when S > (?BS) -><BR>
file:write(Fd, lists:reverse(SoFar)),<BR>
listener(Fd, N, [], 0);<BR>
listener(Fd, N, SoFar, S) -><BR>
receive<BR>
{data, Data} -><BR>
listener(Fd, N + 1, [Data | SoFar],<BR>
S + iolist_size(Data));<BR>
{count, From} -><BR>
From ! {count, N}, listener(Fd, N, SoFar, S);<BR>
stop -> ok;
</BLOCKQUOTE>
</BLOCKQUOTE>
<BLOCKQUOTE TYPE=CITE>
<BLOCKQUOTE>
<BR>
Msg -><BR>
io:format("Unexpected message: ~p~n", [Msg]),<BR>
<BR>
</BLOCKQUOTE>
</BLOCKQUOTE>
<BLOCKQUOTE TYPE=CITE>
<BLOCKQUOTE>
listener(Fd, N, SoFar, S)<BR>
after 0 -><BR>
file:write(Fd, lists:reverse(SoFar)),<BR>
listener(Fd, N, [], 0)<BR>
end.<BR>
<BR>
write(_Data, 0) -> ok;<BR>
write(Data, N) -><BR>
(?MODULE) ! {data, Data}, write(Data, N - 1).<BR>
<BR>
</BLOCKQUOTE>
</BLOCKQUOTE>
<BLOCKQUOTE TYPE=CITE>
<BLOCKQUOTE>
On Fri, Apr 24, 2009 at 4:23 PM, chaitanya Chalasani <<A HREF="mailto:chaitanya.chalasani@gmail.com">chaitanya.chalasani@gmail.com</A>> wrote:<BR>
<BR>
</BLOCKQUOTE>
</BLOCKQUOTE>
<BLOCKQUOTE TYPE=CITE>
<BLOCKQUOTE>
<BLOCKQUOTE>
<BR>
</BLOCKQUOTE>
</BLOCKQUOTE>
</BLOCKQUOTE>
<BLOCKQUOTE TYPE=CITE>
<BLOCKQUOTE>
<BLOCKQUOTE>
Hi,<BR>
<BR>
I am facing performance issue with my file logger. The file logger is an event handler holding the file reference is its state. The performance of the module is way ahead in the smp disabled erl shell compared with default smp enabled erl shell.<BR>
<BR>
I tried ...<BR>
case#1<BR>
<I>Erlang R13B (erts-5.7.1) [source] [rq:1] [async-threads:0] [hipe] [kernel-poll:false]</I><BR>
<BR>
<I>Eshell V5.7.1 (abort with ^G)</I><BR>
<I>1> testLogger:start("test.log").</I><BR>
<I>true</I><BR>
<I>2> testLogger:write("Testing...",1000000).</I><BR>
<I>{ok,done}</I><BR>
<I>3></I><BR>
<BR>
and <BR>
<BR>
case#2<BR>
<I>Erlang R13B (erts-5.7.1) [source] [smp:2:2] [rq:2] [async-threads:0] [hipe] [kernel-poll:false]</I><BR>
<BR>
<I>Eshell V5.7.1 (abort with ^G)</I><BR>
<I>1> testLogger:start("test.log").</I><BR>
<I>true</I><BR>
<I>2> testLogger:write("Testing...",1000000).</I><BR>
<I>{ok,done}</I><BR>
<I>3></I><BR>
<BR>
<BR>
What I observed was -<BR>
Case 1: <BR>
1. The testLogger:write("Testing...",1000000) did not return back the control to the shell until all the messages are written to log.<BR>
2. The data was written at 400 KB/s<BR>
<BR>
Case 2:<BR>
1. The testLogger:write("Testing...",1000000) return back the control to the shell with {ok, done} after a short while but the messages where being written to log even after that.<BR>
2. The data was written at 4 KB/s<BR>
3. I checked in the pman tool it shows current funtion as io:wait_io_mon_reply/2 and the messages in the message pool are about 880k.<BR>
<BR>
Below is the code which I tested.<BR>
<BR>
<I>start(Filename) -> register(testLogger,spawn(testLogger,listener,[Filename])).</I><BR>
<BR>
<I>listener(Filename) -></I><BR>
<I>case file:open(Filename,[append]) of</I><BR>
<I>{ok,Fd} -> listener(Fd,0);</I><BR>
<I>{error,Reason} -> error_logger:error_msg(Reason)</I><BR>
<I>end.</I><BR>
<BR>
<I>listener(Fd,N) -></I><BR>
<I>receive</I><BR>
<I>{data,Data} -> file:write(Fd,Data), listener(Fd,N+1);</I><BR>
<I>{count,From} -> From ! {count,N}, listener(Fd,N);</I><BR>
<I>{stop,From} -> From ! {stop,success}</I><BR>
<I>end.</I><BR>
<BR>
<I>write(_Data,0) -> {ok,done};</I><BR>
<I>write(Data,N) -> testLogger ! {data,Data}, write(Data,N-1).</I><BR>
<BR>
Is there a problem with my code or is it a known issue?<BR>
<BR>
<TABLE CELLSPACING="0" CELLPADDING="0" WIDTH="100%">
<TR>
<TD>
<B><FONT SIZE="2"><FONT COLOR="#808080">------</FONT></FONT></B><BR>
<B><FONT SIZE="2"><FONT COLOR="#808080">CHAITANYA CHALASANI</FONT></FONT></B>
</TD>
</TR>
</TABLE>
</BLOCKQUOTE>
</BLOCKQUOTE>
</BLOCKQUOTE>
<BLOCKQUOTE TYPE=CITE>
<BLOCKQUOTE>
<BLOCKQUOTE>
<BR>
<BR>
</BLOCKQUOTE>
</BLOCKQUOTE>
</BLOCKQUOTE>
<BLOCKQUOTE TYPE=CITE>
<BLOCKQUOTE>
<BLOCKQUOTE>
_______________________________________________<BR>
erlang-questions mailing list<BR>
<A HREF="mailto:erlang-questions@erlang.org">erlang-questions@erlang.org</A><BR>
<A HREF="http://www.erlang.org/mailman/listinfo/erlang-questions">http://www.erlang.org/mailman/listinfo/erlang-questions</A><BR>
<BR>
</BLOCKQUOTE>
</BLOCKQUOTE>
</BLOCKQUOTE>
<BLOCKQUOTE TYPE=CITE>
<BLOCKQUOTE>
<BR>
<BR>
</BLOCKQUOTE>
</BLOCKQUOTE>
<BLOCKQUOTE TYPE=CITE>
<BLOCKQUOTE>
<BR>
</BLOCKQUOTE>
</BLOCKQUOTE>
<BLOCKQUOTE TYPE=CITE>
<BLOCKQUOTE>
<BR>
-- <BR>
--Hynek (Pichi) Vychodil<BR>
<BR>
Analyze your data in minutes. Share your insights instantly. Thrill your boss. Be a data hero!<BR>
Try Good Data now for free: <A HREF="http://www.gooddata.com">www.gooddata.com</A><BR>
<BR>
</BLOCKQUOTE>
</BLOCKQUOTE>
<BLOCKQUOTE TYPE=CITE>
<BR>
<BR>
<BR>
-- <BR>
--Hynek (Pichi) Vychodil<BR>
<BR>
Analyze your data in minutes. Share your insights instantly. Thrill your boss. Be a data hero!<BR>
Try Good Data now for free: <A HREF="http://www.gooddata.com">www.gooddata.com</A><BR>
</BLOCKQUOTE>
<TABLE CELLSPACING="0" CELLPADDING="0" WIDTH="100%">
<TR>
<TD>
<B><FONT SIZE="2"><FONT COLOR="#808080">------</FONT></FONT></B><BR>
<B><FONT SIZE="2"><FONT COLOR="#808080">CHAITANYA CHALASANI</FONT></FONT></B>
</TD>
</TR>
</TABLE>
</BODY>
</HTML>