<html>
<head>
</head>
<body class='hmmessage'><div dir='ltr'>
<div dir="ltr">
<div dir="ltr">
<style><!--
.hmmessage P
{
margin:0px;
padding:0px
}
body.hmmessage
{
font-size: 10pt;
font-family:Tahoma
}
--></style>
<div dir="ltr">Thank you all for taking time to help me out. Some clarification, I've come to the conclusion that gen_server(or more precisely my design) is to blame after running multiple sessions with fprof, eprof and alot of timer:now_diff and timer:tc. For example, if I put load on the system and time my <span style="background-color: rgb(255, 255, 255); white-space: pre-wrap; font-size: 10pt; ">handle_call({long_operation,Data},From,State) I see that the execution time begins rise from <10ms to around ~2000ms and stabilizing there under my load. If I remove the last gen_server and instead implement it as a simple function it stabilizes around ~1000ms and removing yet another one gives me yet lower execution times.</span></div><div dir="ltr"><span style="white-space: pre-wrap;"><br></span></div><div dir="ltr"><span style="white-space: pre-wrap;">A word on my design, a lot of the time the only reason we use a gen_server is for keeping a state and more precisely a connection. For instance a generic db backend which keeps the pid of the db connection, a cache backend which keeps the memcache connection pid, my own server that keep state. and within my </span><span style="background-color: rgb(255, 255, 255); white-space: pre-wrap; font-size: 10pt; ">handle_call({long_operation,Data},From,State) I could be doing a couple of gen_server:call's and within the database gen_server I could be doing additional gen_server call's to the cache backend. Why we've constructed it as separate gen_server's is also to be able to call it from different parts of the system like db:do_something() or cache:get().</span></div><div dir="ltr"><span style="background-color: rgb(255, 255, 255); white-space: pre-wrap; font-size: 10pt; "><br></span></div><div dir="ltr"><span style="background-color: rgb(255, 255, 255); "><font size="2"><span style="white-space: pre-wrap;">On paper this looked like a clean and nice design but under load it starts to behave </span></font><span style="white-space: pre-wrap;">poorly</span><font size="2"><span style="white-space: pre-wrap;">.</span></font></span></div><div dir="ltr"><span style="white-space: pre-wrap;"><br></span></div><div dir="ltr"><div>To answer all in order:</div><div># Gianfranco Alongi 2012-01-28:</div><div>> You could use the synchronized serialization to generate push-back<br>> behaviour from your system,<br>> so that you do not handle a new request before it's possible - maybe<br>> you are already doing this, or not.<br>I don' understand what this mean. Can you please clarify.</div><div><br></div><div># Matthew Evans 2012-01-28</div><div><div><span style="font-size: 10pt; ">> The obvious, and maybe non-OTP, answer is to hold some of this state information in a public or protected named ETS table</span></div><div><span style="font-size: 10pt; ">> that your clients read from directly. A single gen_server can still own and write to that ETS table.</span></div><div><span style="font-size: 10pt; ">Sounds like a smart approach.</span></div><div><br></div><div>> Another obvious answer is to provide back-pressure of some kind to prevent clients from requesting data when it is under load.</div><div>I don't understand this fully.<br><br>> You might find that a particular infrequent gen_server:call operation is taking a long time to complete causing a message queue</div><div>> to suddenly grow. You might want to change such an operation from:</div><div>I've done that on the first most handle_call. That's what I meant with deferring gen_server responses. I saw some speedups on the first handle_call but doing this on short lived handle_call's did see slowdowns.</div><div><br></div><div># Jesper Louis Andersen 2012-01-28</div><div>> This would be my first idea. Create an ETS table being protected. Writes to the table goes through the gen_server,<br></div><div>I like this approach, not as "beautiful" as a pure OTP-approach, but if does the trick. Hey.</div><div><br></div># Jachym Holecek 2012-01-29</div><div>> <span style="background-color: rgb(255, 255, 255); white-space: pre-wrap; font-size: 10pt; ">You're probably treating asynchronous things as synchronous somewhere along </span><span style="background-color: rgb(255, 255, 255); white-space: pre-wrap; font-size: 10pt; ">the path, inflicting collateral damage to concurrent users?</span></div><div><span style="white-space: pre-wrap;">it's basically a get_data() function where we need to do roundtrips to db, other stuff to be able to return the value. Can I do this in a asynchronous fashion? I tend to think of get's as handle_calls and sets as hanlde_cast.</span></div><div><span style="white-space: pre-wrap;"><br></span></div><div><span style="white-space: pre-wrap;">Big thank you all.</span></div><div><span style="white-space: pre-wrap;"><br></span></div><div><span style="white-space: pre-wrap;">/dang</span></div><div><span style="white-space: pre-wrap;"><br></span><div><div id="SkyDrivePlaceholder"></div>> Date: Sat, 28 Jan 2012 18:13:25 -0500<br>> From: freza@circlewave.net<br>> To: jesper.louis.andersen@erlang-solutions.com<br>> CC: erlang-questions@erlang.org<br>> Subject: Re: [erlang-questions] Slow when using chained gen_server:call's, redesign or optimize?<br>> <br>> Hi,<br>> <br>> [Replying to multiple replies at once, all quoted text reformatted for<br>> readability (seems people these days can't be bothered!?).]<br>> <br>> [Warning: excessively nitpicky content ahead, here and there.]<br>> <br>> # God Dang 2012-01-28:<br>> > I'm creating a system where I've ended up with alot of gen_servers that<br>> > provides a clean interface. When I run this under load I see that the<br>> > gen_server:call's is becoming a bottleneck.<br>> <br>> You're probably treating asynchronous things as synchronous somewhere along<br>> the path, inflicting collateral damage to concurrent users? Synchronous<br>> high-level API is fine, but if you know some operation is expensive or<br>> depends on response from the outside world you should record request<br>> context in ETS (row looking something like {Req_id, Timeout, From[, ...]}<br>> and process requests out-of-order OR offload the actual processing to<br>> short lived worker processes like Matthew Evans says OR a combination<br>> of both OR somesuch.<br>> <br>> My point being that gen_server:call/N, by itself, is *very* fast in<br>> practice, so chances are you're doing something wrong elsewhere.<br>> <br>> Other (unlikely) thing: you're not sending very large data structures in<br>> messages, are you? That could hurt, but there are ways to address that<br>> too if needed.<br>> <br>> # Matthew Evans 2012-01-28:<br>> > Another obvious answer is to provide back-pressure of some kind to prevent<br>> > clients from requesting data when it is under load.<br>> <br>> On external interfaces (or for global resource usage of some sort): yes, a<br>> fine idea (a clear "must have", actually!); but doing this internally would<br>> seem excessively defensive to me, unless further justification was given.<br>> <br>> > You might want to change such an operation from:<br>> ><br>> > handle_call({long_operation,Data},From,State) -><br>> > Rsp = do_lengthy_operation(Data),<br>> > {reply, Rsp, State};<br>> ><br>> > to:<br>> ><br>> > handle_call({long_operation,Data},From,State) -><br>> > spawn(fun() -> Rsp = do_lengthy_operation(Data), gen_server:reply(Rsp,From) end),<br>> > {noreply, State};<br>> <br>> 1. Why do people bother introducing "one-shot" variables for trivial<br>> expressions they could have inlined? Means less context to maintain<br>> when reading the code...<br>> <br>> 2. Surely you meant proc_lib:spawn_link/X there, didn't you? SASL logs<br>> and fault propagation are the reason. While there are exceptions to<br>> this, they're extremely rare.<br>> <br>> 3. The order of arguments to gen_server:reply/2 is wrong.<br>> <br>> Regarding the general approach: yes, a fine idea too. Depending on what<br>> do_lengthy_operation/1 does putting these workers under supervisor might<br>> be called for.<br>> <br>> # Jesper Louis Andersen 2012-01-28:<br>> > This would be my first idea. Create an ETS table being protected.<br>> > Writes to the table goes through the gen_server,<br>> <br>> Yes, a fine idea too -- ETS is one of the less obvious cornerstones<br>> of Erlang programming (but don't tell "purity" fascists )... One<br>> detail: almost all of my ETS tables are public even when many of<br>> them are really treated as private or protected, reason is to keep<br>> high degree of runtime tweakability just in case (this might be a<br>> bit superstitious I admit).<br>> <br>> > -export([write/1, read/1]).<br>> > <br>> > write(Obj) -><br>> > call({write, Obj}).<br>> > <br>> > call(M) -><br>> > gen_server:call(?SERVER, M, infinity).<br>> <br>> 1. Abstracting trivial functionality such as call/1 above only<br>> obfuscates code for precisely zero gain.<br>> <br>> 2. Same goes for typing "?SERVER" instead of the actual server<br>> name. Using "?MODULE" is however alright, as long as it's<br>> only referred to from current module (as it should).<br>> <br>> 3. No infinite timeouts without very good justification! You're<br>> sacrificing a good default protective measure for no good<br>> reason...<br>> <br>> > but reads happen in the calling process of the API and does not go<br>> > through the gen_server at all,<br>> > <br>> > read(Key) -><br>> > case ets:lookup(?TAB, Key) of<br>> > [] -> not_found;<br>> > [_|_] = Objects -> {ok, Objects}<br>> > end.<br>> <br>> (2) from above also applies to "?TAB" here. More to the point, it's<br>> sometimes perfectly OK to do table writes directly from caller's<br>> context too, like:<br>> <br>> write(Item) -><br>> true = ets:insert_new(actual_table_name, Item).<br>> <br>> It can of course be very tricky business and needs good thinking first.<br>> I bet you're aware of this, mentioning it just because it's a handy<br>> trick that doesn't seem to be widely known.<br>> <br>> > Creating the table with {read_concurrency, true} as the option will<br>> > probably speed up reads by quite a bit as well. It is probably going<br>> > to be a lot faster than having all caching reads going through that<br>> > single point of contention. Chances are that just breaking parts of<br>> > the chain is enough to improve the performance of the system.<br>> <br>> Well, yes, avoiding central points of contention (such as blessed<br>> processes or, you guessed it, ETS tables) is certainly good engineering<br>> practice, but see first part of this email for other considerations.<br>> <br>> BR,<br>> -- Jachym<br>> _______________________________________________<br>> erlang-questions mailing list<br>> erlang-questions@erlang.org<br>> http://erlang.org/mailman/listinfo/erlang-questions<br></div></div></div>
</div>
</div>
</div></body>
</html>