Hi List,<br><br>I'm a bit late replying to this thread, only reading it now. Probably too late to help the original poster but here's my contribution anyway.<br><br>Matthew, hope you don't mind me suggesting a minor improvement to your code snippet, just to demonstrate a larger point. Instead of:<br>
<br>handle_call({long_operation,Data},From,State) -><br> spawn(fun() -><br> Rsp = do_lengthy_operation(Data),<br> gen_server:reply(Rsp,From)<br> end),<br> {noreply, State};<br><br>I prefer getting the calling thread to do as much work as possible.<br>
Here a process is spawned to do the work and then die, all while the calling process waits. I think it's preferable to get the calling process to do the work itself, it avoids the spawn and any crashes in the do_lengthy_operation are argubly easier to debug. Eg:<br>
<br>long_operation(Data) -><br> State = gen_server:call(?MODULE, get_state),<br> do_lengthy_operation(Data,State).<br><br>handle_call(get_state,_From,State) -><br> {reply,State,State};<br><br>Normally when I'm writing a new gen_server I get the calling process to do as much work as possible and try to get the handle_calls/casts to only do the work that requires mutual exclusion. I normally store what I can in a protected, named_table ets table so that interface funtions that need this can read it without going through the gen_server but I do all updates in a handle_call/cast.<br>
<br>Once one of my gen_servers became a bottle-neck in a system and I ended up taking these steps to fix the problem. Since then I've adopted this as a rule of thumb when writing new gen_servers and avoid such pointless bottle-necks that I'd have to fix later on anyway.<br>
<br>I inherited some code with the same problem as the original poster with gen_server calls going five-deep in one instance. Everything worked fine at low load levels but once the load crossed a threshold there was no recovery as every management process spent most of its time waiting for other management processes to reply while their own message queues built up. Meanwhile the worker threads would timeout, reconnect and add to the backlog. The message queues just built up until the entire node fell over.<br>
<br>After making sure that the chained calls weren't required to keep the data consistent I moved as work as possible to the calling process. This way I kept the interface unchanged for each module but avoided the case where each process in the chain was tied up, waiting for the last one to complete.<br>
<br>I think for newbies it's important to be aware of which process is doing the work. I know when I started using Erlang it took a while to get my head around the concurrency.<br><br>//TTom.<br><br><br><br><div class="gmail_quote">
On Sat, Jan 28, 2012 at 3:18 PM, Matthew Evans <span dir="ltr"><<a href="mailto:mattevans123@hotmail.com">mattevans123@hotmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div><div dir="ltr">
Of course you need to run a profiler such as fprof to see what's going on.<div><br></div><div>Sounds like a classic head of line blocking problem. Many requests, possibly from processes on different schedulers/cores, all getting serialized on a single gen_server.</div>
<div><br></div><div>The obvious, and maybe non-OTP, answer is to hold some of this state information in a public or protected named ETS table that your clients read from directly. A single gen_server can still own and write to that ETS table. </div>
<div><br></div><div>Another obvious answer is to provide back-pressure of some kind to prevent clients from requesting data when it is under load.<br><br>You might find that a particular infrequent gen_server:call operation is taking a long time to complete causing a message queue to suddenly grow. You might want to change such an operation from:</div>
<div><br></div><div>handle_call({long_operation,Data},From,State) -></div><div> Rsp = do_lengthy_operation(Data),</div><div> {reply, Rsp, State};</div><div><br></div><div>to:</div><div><br></div><div><div>handle_call({long_operation,Data},From,State) -></div>
<div> spawn(fun() -></div><div> Rsp = do_lengthy_operation(Data),</div><div> gen_server:reply(Rsp,From)</div><div> end), </div><div> {noreply, State};</div><div><br></div><div class="hm HOEnZb">
<div><br></div></div><div><div class="hm HOEnZb"><div></div><hr>From: <a href="mailto:goddang@hotmail.com" target="_blank">goddang@hotmail.com</a><br>To: <a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a><br>
Date: Sat, 28 Jan 2012 00:06:04 +0000<br>Subject: [erlang-questions] Slow when using chained gen_server:call's, redesign or optimize?</div><div><div class="h5"><br><br>
<div dir="ltr">
I'm creating a system where I've ended up with alot of gen_servers that provides a clean interface. When I run this under load I see that the gen_server:call's is becoming a bottleneck.<div>For instance, In a handle_request I might ask an other gen_server to get me a cached object, then ask the database something, then etc...</div>
<div>and in some cases I have my-gen_server->cache-gen_server->memcache-client-gen_server as you see it stacks up to alot of steps. I've tried to optimize with deferring gen_server responses and that has given a slight performance improvement but not as drastical as if I for instance bypass one gen_server instance.</div>
<div><br></div><div>Is there a better way to go about this or some smart optimization to do? And FYI, I use gen_server when I need to keep a state of a connection or something so if the answer is to scrap or reduce the number of gen_servers I will need to keep those connections somewhere else.</div>
<div><br></div><div>Thanks, Dang</div><div><br></div> </div>
<br></div></div><div class="im">_______________________________________________
erlang-questions mailing list
<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a>
<a href="http://erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a></div></div></div> </div></div>
<br>_______________________________________________<br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org">erlang-questions@erlang.org</a><br>
<a href="http://erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
<br></blockquote></div><br>