<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div dir="ltr"><meta http-equiv="content-type" content="text/html; charset=utf-8"><div dir="ltr">This function nothing is but a single postgres query, a “select * from users where id = ‘123’”, properly indexed.</div><div dir="ltr"><br></div><div dir="ltr">The only thing I can see is a latency towards the db (infra-aws regions unfortunately). It really is that at a certain moment, randomly (sometimes after 5 minutes, other times after 2 days) this happens and there’s no recovery whatsoever. </div><div dir="ltr"><br></div><div dir="ltr">I could start rejecting http calls if the poolboy can’t perform a checkout, not sure how to do that (need to check the library better). </div><div dir="ltr"><br><blockquote type="cite">On 3 Dec 2019, at 19:26, Jesper Louis Andersen <jesper.louis.andersen@gmail.com> wrote:<br><br></blockquote></div><blockquote type="cite"><div dir="ltr"><div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">The keys_db function is currently executing gen:do_call/4</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">It is blocked on some other process somewhere, waiting for it to respond, so it can respond back to the 11906 requests it has :) Chances are that the next request in the queue will do a gen:do_call/4 for the 11905 there are left in the queue. And in the mean time, 30 new requests arrived, so we are now at 11935.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">In short, it is likely something is running too slow to handle the current load on the system and furthermore, there is no flow control in the system to make the callers reduce the load.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Since it happens randomly, it is likely there is a single large request, done by a single user now and then. And this request stalls the system. Probably enough to have the queue grow and errors start happening. Put a limit on what can be requested, and force the client to cooperate by having it call in with a "plz continue" token.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">You can also ask the system for a more detailed stack trace, see erlang:process_info/2 and current_stacktrace when it goes bad. This can often tell you which gen_server call is being made and to whom, narrowing down the problem.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Dec 3, 2019 at 7:14 PM Roberto Ostinelli <<a href="mailto:ostinelli@gmail.com">ostinelli@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Forgot to mention: this happens in a completely random way.</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Dec 3, 2019 at 6:48 PM Roberto Ostinelli <<a href="mailto:ostinelli@gmail.com" target="_blank">ostinelli@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><p style="box-sizing:border-box;margin-bottom:16px;color:rgb(36,41,46);font-family:-apple-system,system-ui,"Segoe UI",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji";font-size:14px;margin-top:0px">All,<br style="box-sizing:border-box">I ame experiencing the following error when calling a transaction in poolboy as per the README:</p><div style="box-sizing:border-box;margin-bottom:16px;color:rgb(36,41,46);font-family:-apple-system,system-ui,"Segoe UI",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji";font-size:14px;overflow:visible"><pre style="box-sizing:border-box;font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,monospace;font-size:11.9px;margin-top:0px;margin-bottom:0px;padding:16px;overflow:auto;line-height:1.45;background-color:rgb(246,248,250);border-radius:3px;word-break:normal"><span style="box-sizing:border-box;color:rgb(111,66,193)">equery</span>(<span style="box-sizing:border-box">PoolName</span>, <span style="box-sizing:border-box">Stmt</span>, <span style="box-sizing:border-box">Params</span>) <span style="box-sizing:border-box;color:rgb(215,58,73)">-></span>
<span style="box-sizing:border-box;color:rgb(111,66,193)">poolboy</span>:<span style="box-sizing:border-box;color:rgb(111,66,193)">transaction</span>(<span style="box-sizing:border-box">PoolName</span>, <span style="box-sizing:border-box;color:rgb(215,58,73)">fun</span>(<span style="box-sizing:border-box">Worker</span>) ->
<span style="box-sizing:border-box;color:rgb(111,66,193)">gen_server</span>:<span style="box-sizing:border-box;color:rgb(111,66,193)">call</span>(<span style="box-sizing:border-box">Worker</span>, {<span style="box-sizing:border-box;color:rgb(0,92,197)">equery</span>, <span style="box-sizing:border-box">Stmt</span>, <span style="box-sizing:border-box">Params</span>})
<span style="box-sizing:border-box;color:rgb(215,58,73)">end</span>).</pre></div><pre style="box-sizing:border-box;font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,monospace;font-size:11.9px;margin-top:0px;margin-bottom:16px;padding:16px;overflow:auto;line-height:1.45;background-color:rgb(246,248,250);border-radius:3px;color:rgb(36,41,46)"><code style="box-sizing:border-box;font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,monospace;font-size:11.9px;padding:0px;margin:0px;background:initial;border-radius:3px;word-break:normal;border:0px;display:inline;overflow:visible;line-height:inherit">** exception exit: {timeout,{gen_server,call,
[keys_db,{checkout,#Ref<0.0.1.156295>,true},5000]}}
in function gen_server:call/3 (gen_server.erl, line 212)
in call from poolboy:checkout/3 (/home/ubuntu/workspace/myapp/_build/default/lib/poolboy/src/poolboy.erl, line 55)
in call from poolboy:transaction/3 (/home/ubuntu/workspace/myapp/_build/default/lib/poolboy/src/poolboy.erl, line 74)
</code></pre><p style="box-sizing:border-box;margin-top:0px;margin-bottom:16px;color:rgb(36,41,46);font-family:-apple-system,system-ui,"Segoe UI",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji";font-size:14px">The process queue keeps on increasing, and I can see the following:</p><pre style="box-sizing:border-box;font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,monospace;font-size:11.9px;margin-top:0px;margin-bottom:16px;padding:16px;overflow:auto;line-height:1.45;background-color:rgb(246,248,250);border-radius:3px;color:rgb(36,41,46)"><code style="box-sizing:border-box;font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,monospace;font-size:11.9px;padding:0px;margin:0px;background:initial;border-radius:3px;word-break:normal;border:0px;display:inline;overflow:visible;line-height:inherit">3> erlang:process_info(whereis(keys_db)).
[{registered_name,keys_db},
{current_function,{gen,do_call,4}},
{initial_call,{proc_lib,init_p,5}},
{status,waiting},
{message_queue_len,11906},
{messages,[{'$gen_cast',{cancel_waiting,#Ref<0.0.1.138090>}},
{'$gen_call',{<0.15224.0>,#Ref<0.0.1.139621>},
{checkout,#Ref<0.0.1.139620>,true}},
{'$gen_call',{<0.15139.0>,#Ref<0.0.1.139649>},
{checkout,#Ref<0.0.1.139648>,true}},
{'$gen_cast',{cancel_waiting,#Ref<0.0.1.138159>}},
{'$gen_cast',{cancel_waiting,#Ref<0.0.1.138175>}},
{'$gen_cast',{cancel_waiting,#Ref<0.0.1.138232>}},
{'$gen_cast',{cancel_waiting,#Ref<0.0.1.138252>}},
{'$gen_cast',{cancel_waiting,#Ref<0.0.1.138261>}},
{'$gen_cast',{cancel_waiting,#Ref<0.0.1.138286>}},
{'$gen_call',{<0.15235.0>,#Ref<0.0.1.139774>},
{checkout,#Ref<0.0.1.139773>,true}},
{'$gen_cast',{cancel_waiting,#Ref<0.0.2.77777>}},
{'$gen_cast',{cancel_waiting,#Ref<0.0.1.138318>}},
{'$gen_cast',{cancel_waiting,#Ref<0.0.1.138336>}},
{'$gen_call',{<0.15233.0>,#Ref<0.0.1.139816>},
{checkout,#Ref<0.0.1.139815>,true}},
{'$gen_call',{<0.15245.0>,#Ref<0.0.1.139854>},
{checkout,#Ref<0.0.1.139853>,true}},
{'$gen_call',{<0.15237.0>,#Ref<0.0.2.78173>},
{checkout,#Ref<0.0.2.78172>,...}},
{'$gen_cast',{cancel_waiting,#Ref<0.0.1.138407>}},
{'$gen_call',{<0.15228.0>,...},{...}},
{'$gen_call',{...},...},
{'$gen_call',...},
{...}|...]},
{links,[<0.714.1>,<0.817.1>,<0.947.1>,<0.1015.1>,<0.1045.1>,
<0.1048.1>,<0.1038.1>,<0.983.1>,<0.1002.1>,<0.962.1>,
<0.877.1>,<0.909.1>,<0.938.1>,<0.892.1>,<0.849.1>,<0.866.1>,
<0.832.1>,<0.765.1>,<0.789.1>,<0.804.1>|...]},
{dictionary,[{'$initial_call',{poolboy,init,1}},
{'$ancestors',[pgpool_sup,<0.673.0>]}]},
{trap_exit,true},
{error_handler,error_handler},
{priority,normal},
{group_leader,<0.672.0>},
{total_heap_size,393326},
{heap_size,196650},
{stack_size,33},
{reductions,14837255},
{garbage_collection,[{max_heap_size,#{error_logger => true,
kill => true,
size => 0}},
{min_bin_vheap_size,46422},
{min_heap_size,233},
{fullsweep_after,10},
{minor_gcs,3}]},
{suspending,[]}]
</code></pre><p style="box-sizing:border-box;margin-top:0px;margin-bottom:16px;color:rgb(36,41,46);font-family:-apple-system,system-ui,"Segoe UI",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji";font-size:14px">Does someone have an insight of what may be going wrong? I see that the process status is <code style="box-sizing:border-box;font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,monospace;font-size:11.9px;padding:0.2em 0.4em;margin:0px;background-color:rgba(27,31,35,0.05);border-radius:3px">waiting</code>...</p><p style="box-sizing:border-box;margin-top:0px;color:rgb(36,41,46);font-family:-apple-system,system-ui,"Segoe UI",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji";font-size:14px;margin-bottom:0px">Thank you,<br style="box-sizing:border-box">r.</p></div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature">J.</div>
</div></blockquote></div></body></html>