<div dir="ltr"><div><div><div>Hi,<br><br></div>I've discovered a memory leak in <span style="background:none repeat scroll 0% 0% yellow" class="">httpc</span>, which occurs when you are using <span style="background:none repeat scroll 0% 0% yellow" class="">httpc</span> with a stand_alone profile and using persistent http connections in R16B03-1.<br>
<br></div>After some investigation, it turns out that the issue has been around since R14B04 (maybe even before), but it has been hidden by another bug which was fixed somewhere in R15.<br><br></div><div></div><div>Synopsis:<br>
--------------<br><br></div><div>1. Start a stand_alone profile.<br></div><div>2. Establish a persistent http connection to a server that is never recycled.<br></div><div>3. In a continuous loop, keep on submitting requests on the same persistent connection at an interval less than <span class="">keep_alive_timeout (default is 2 minutes), i.e. the <span style="background:none repeat scroll 0% 0% yellow" class="">tcp</span> connection is kept alive (assuming the server does not kill the tcp connection).<br>
</span></div><div><span class="">4. The <span style="background:none repeat scroll 0% 0% yellow" class="">ETS</span> table used by the <span style="background:none repeat scroll 0% 0% yellow" class="">httpc</span>_manager to keep track of requests per <span style="background:none repeat scroll 0% 0% yellow" class="">httpc</span>_handler process (<span style="background:none repeat scroll 0% 0% yellow" class="">PROFILENAME</span>__handler_db) grows without ever being cleaned up, even though the <span style="background:none repeat scroll 0% 0% yellow" class="">httpc</span>_handler sends "request_done" notifications (gen_server:cast) for each completed request, back to the <span style="background:none repeat scroll 0% 0% yellow" class="">httpc</span>_manager.<br>
</span></div><div><span class="">5. Either kill the server endpoint or stop submitting requests for longer than </span><span class=""><span class="">keep_alive_timeout, </span>so that the <span style="background:none repeat scroll 0% 0% yellow" class="">tcp</span> connection in 2. can be recycled, either way, the <span style="background:none repeat scroll 0% 0% yellow" class="">httpc</span>_manager process will get an 'EXIT' / 'DOWN' notification, and THEN ONLY it will clean the </span><span class=""><span class=""><span style="background:none repeat scroll 0% 0% yellow" class="">PROFILENAME</span></span>__handler_db <span style="background:none repeat scroll 0% 0% yellow" class="">ETS</span> table.<br>
</span></div><div><span class=""><br></span></div><div>Code to reproduce (incl. workaround) in R16B03-1:<br>-------------------------------------------------------------------------<br>-module(reproduce_<span style="background:none repeat scroll 0% 0% yellow" class="">httpc</span>_stand_alone_memory_leak_r16b03_1).<br>
<br>-export([go/0]).<br><br>go() -><br><br> application:start(<span style="background:none repeat scroll 0% 0% yellow" class="">inets</span>),<br><br> <span style="background:none repeat scroll 0% 0% yellow" class="">TcpPort</span> = 20031,<br>
<span style="background:none repeat scroll 0% 0% yellow" class="">NumClientRequests</span> = 5,<br><br>%%%%%%%%%%%%%%%%%%%%%%%%%%%%%<br>%Start a <span style="background:none repeat scroll 0% 0% yellow" class="">httpc</span> server<br>
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%<br> <span style="background:none repeat scroll 0% 0% yellow" class="">NotifyReadyPid</span> = self(),<br> <span style="background:none repeat scroll 0% 0% yellow" class="">ServerPid</span> = spawn(fun() -> server_listen(<span style="background:none repeat scroll 0% 0% yellow" class="">NotifyReadyPid</span>, <span style="background:none repeat scroll 0% 0% yellow" class="">TcpPort</span>) end),<br>
receive listening -> <span style="background:none repeat scroll 0% 0% yellow" class="">ok</span> end,<br>%%%%%%%%%%%%%%%%%%%%%%%%%%%%%<br><br>%%%%%%%%%%%%%%%%%%%%%%%%%%%%%<br>%Start the <span style="background:none repeat scroll 0% 0% yellow" class="">httpc</span> stand_alone profile<br>
{<span style="background:none repeat scroll 0% 0% yellow" class="">ok</span>, <span style="background:none repeat scroll 0% 0% yellow" class="">ProfilePid</span>} = <span style="background:none repeat scroll 0% 0% yellow" class="">inets</span>:start(<span style="background:none repeat scroll 0% 0% yellow" class="">httpc</span>, [{profile, my_profile}], stand_alone),<br>
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%<br><br>%%%%%%%%%%%%%%%%%%%%%%%%%%%%%<br>%First run without workaround<br> <span style="background:none repeat scroll 0% 0% yellow" class="">io</span>:format("<span style="background:none repeat scroll 0% 0% yellow" class="">whereis</span>(stand_alone_my_profile) => ~p~n", [<span style="background:none repeat scroll 0% 0% yellow" class="">whereis</span>(stand_alone_my_profile)]),<br>
<span style="background:none repeat scroll 0% 0% yellow" class="">ClientPid</span> = start_the_client(<span style="background:none repeat scroll 0% 0% yellow" class="">ProfilePid</span>, <span style="background:none repeat scroll 0% 0% yellow" class="">TcpPort</span>, <span style="background:none repeat scroll 0% 0% yellow" class="">NumClientRequests</span>),<br>
monitor(process, <span style="background:none repeat scroll 0% 0% yellow" class="">ClientPid</span>),<br> receive {'DOWN', _, _, <span style="background:none repeat scroll 0% 0% yellow" class="">ClientPid</span>, _} -> <span style="background:none repeat scroll 0% 0% yellow" class="">ok</span> end,<br>
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%<br><br><br>%%%%%%%%%%%%%%%%%%%%%%%%%%%%%<br>%<span style="background:none repeat scroll 0% 0% yellow" class="">HttpClient</span> connection will recycle <br>%after 1000ms of inactivity, <span style="background:none repeat scroll 0% 0% yellow" class="">which</span> will trigger the<br>
%cleanup of the <span style="background:none repeat scroll 0% 0% yellow" class="">ETS</span> table<br> timer:sleep(1100), 0 = <span style="background:none repeat scroll 0% 0% yellow" class="">ets</span>:info(stand_alone_my_profile__handler_db, size),<br>
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%<br><br>%%%%%%%%%%%%%%%%%%%%%%%%%%%%%<br>%Implement workaround <br> <span style="background:none repeat scroll 0% 0% yellow" class="">io</span>:format("now for the workaround... register(~p,~p) ~n",[stand_alone_my_profile, <span style="background:none repeat scroll 0% 0% yellow" class="">ProfilePid</span>]),<br>
register(stand_alone_my_profile, <span style="background:none repeat scroll 0% 0% yellow" class="">ProfilePid</span>),<br>%%%%%%%%%%%%%%%%%%%%%%%%%%%%%<br><br>%%%%%%%%%%%%%%%%%%%%%%%%%%%%%<br>%Second run with workaround<br>
<span style="background:none repeat scroll 0% 0% yellow" class="">io</span>:format("<span style="background:none repeat scroll 0% 0% yellow" class="">whereis</span>(stand_alone_my_profile) => ~p~n", [<span style="background:none repeat scroll 0% 0% yellow" class="">whereis</span>(stand_alone_my_profile)]),<br>
ClientPid2 = start_the_client(<span style="background:none repeat scroll 0% 0% yellow" class="">ProfilePid</span>, <span style="background:none repeat scroll 0% 0% yellow" class="">TcpPort</span>, <span style="background:none repeat scroll 0% 0% yellow" class="">NumClientRequests</span>),<br>
monitor(process, ClientPid2),<br> receive {'DOWN', _, _, ClientPid2, _} -> <span style="background:none repeat scroll 0% 0% yellow" class="">ok</span> end,<br>%%%%%%%%%%%%%%%%%%%%%%%%%%%%%<br><br> <span style="background:none repeat scroll 0% 0% yellow" class="">init</span>:stop().<br>
<br><br>%<span style="background:none repeat scroll 0% 0% yellow" class="">HttpServer</span><br>server_listen(<span style="background:none repeat scroll 0% 0% yellow" class="">NotifyReadyPid</span>, Port) -><br> {<span style="background:none repeat scroll 0% 0% yellow" class="">ok</span>, LS} = gen_<span style="background:none repeat scroll 0% 0% yellow" class="">tcp</span>:listen(Port, [{<span style="background:none repeat scroll 0% 0% yellow" class="">ip</span>, {127,0,0,1}}, binary]),<br>
<span style="background:none repeat scroll 0% 0% yellow" class="">NotifyReadyPid</span> ! listening,<br> server_accept(LS).<br><br>server_accept(LS) -><br> {<span style="background:none repeat scroll 0% 0% yellow" class="">ok</span>, AS} = gen_<span style="background:none repeat scroll 0% 0% yellow" class="">tcp</span>:accept(LS),<br>
<span style="background:none repeat scroll 0% 0% yellow" class="">ok</span> = <span style="background:none repeat scroll 0% 0% yellow" class="">inet</span>:<span style="background:none repeat scroll 0% 0% yellow" class="">setopts</span>(AS, [{packet, http}, {active, false}]),<br>
<span style="background:none repeat scroll 0% 0% yellow" class="">WPid</span> = spawn(fun() -> receive <span style="background:none repeat scroll 0% 0% yellow" class="">gogogo</span> -> <span style="background:none repeat scroll 0% 0% yellow" class="">ok</span> end, server_worker(AS) end),<br>
<span style="background:none repeat scroll 0% 0% yellow" class="">ok</span> = gen_<span style="background:none repeat scroll 0% 0% yellow" class="">tcp</span>:controlling_process(AS, <span style="background:none repeat scroll 0% 0% yellow" class="">WPid</span>),<br>
<span style="background:none repeat scroll 0% 0% yellow" class="">WPid</span> ! <span style="background:none repeat scroll 0% 0% yellow" class="">gogogo</span>,<br> server_accept(LS).<br><br>server_worker(AS) -><br>
case <span style="background:none repeat scroll 0% 0% yellow" class="">recv</span>_to_http_<span style="background:none repeat scroll 0% 0% yellow" class="">eoh</span>(AS) of<br> <span style="background:none repeat scroll 0% 0% yellow" class="">ok</span> -><br>
<span style="background:none repeat scroll 0% 0% yellow" class="">Rsp</span> = <<"HTTP/1.1 200 OK\r\nContent-Length: 2\r\nContent-Type: text/html\r\n\r\nOK">>,<br> <span style="background:none repeat scroll 0% 0% yellow" class="">ok</span> = gen_<span style="background:none repeat scroll 0% 0% yellow" class="">tcp</span>:send(AS, <span style="background:none repeat scroll 0% 0% yellow" class="">Rsp</span>),<br>
server_worker(AS);<br> closed -><br> <span style="background:none repeat scroll 0% 0% yellow" class="">ok</span><br> end.<br><br><span style="background:none repeat scroll 0% 0% yellow" class="">recv</span>_to_http_<span style="background:none repeat scroll 0% 0% yellow" class="">eoh</span>(AS) -><br>
case gen_<span style="background:none repeat scroll 0% 0% yellow" class="">tcp</span>:<span style="background:none repeat scroll 0% 0% yellow" class="">recv</span>(AS, 0) of<br> {<span style="background:none repeat scroll 0% 0% yellow" class="">ok</span>, http_<span style="background:none repeat scroll 0% 0% yellow" class="">eoh</span>} -><br>
<span style="background:none repeat scroll 0% 0% yellow" class="">ok</span>;<br> {error, closed} -><br> closed;<br> {<span style="background:none repeat scroll 0% 0% yellow" class="">ok</span>, _Packet} -><br>
<span style="background:none repeat scroll 0% 0% yellow" class="">recv</span>_to_http_<span style="background:none repeat scroll 0% 0% yellow" class="">eoh</span>(AS)<br> end.<br><br>%<span style="background:none repeat scroll 0% 0% yellow" class="">HttpClient</span><br>
start_the_client(<span style="background:none repeat scroll 0% 0% yellow" class="">ProfilePid</span>, <span style="background:none repeat scroll 0% 0% yellow" class="">TcpPort</span>, <span style="background:none repeat scroll 0% 0% yellow" class="">NumRequests</span>) -><br>
<span style="background:none repeat scroll 0% 0% yellow" class="">ok</span> = <span style="background:none repeat scroll 0% 0% yellow" class="">httpc</span>:set_options([{max_sessions, 1}, {max_pipeline_length, 0}, {pipeline_timeout, 0}, {max_keep_alive_length, 1}, {keep_alive_timeout, 1000}], <span style="background:none repeat scroll 0% 0% yellow" class="">ProfilePid</span>),<br>
<span style="background:none repeat scroll 0% 0% yellow" class="">Url</span> = "http://<span style="background:none repeat scroll 0% 0% yellow" class="">localhost</span>:"++integer_to_list(<span style="background:none repeat scroll 0% 0% yellow" class="">TcpPort</span>)++"/temp",<br>
spawn(fun() -> worker_thread(<span style="background:none repeat scroll 0% 0% yellow" class="">Url</span>, <span style="background:none repeat scroll 0% 0% yellow" class="">ProfilePid</span>, <span style="background:none repeat scroll 0% 0% yellow" class="">NumRequests</span>) end).<br>
<br>worker_thread(<span style="background:none repeat scroll 0% 0% yellow" class="">Url</span>, <span style="background:none repeat scroll 0% 0% yellow" class="">ProfilePid</span>, 0) -><br> <span style="background:none repeat scroll 0% 0% yellow" class="">io</span>:format("<span style="background:none repeat scroll 0% 0% yellow" class="">ets</span>:info(stand_alone_my_profile__handler_db, info) => ~p~n", [<span style="background:none repeat scroll 0% 0% yellow" class="">ets</span>:info(stand_alone_my_profile__handler_db, size)]);<br>
worker_thread(<span style="background:none repeat scroll 0% 0% yellow" class="">Url</span>, <span style="background:none repeat scroll 0% 0% yellow" class="">ProfilePid</span>, <span style="background:none repeat scroll 0% 0% yellow" class="">NumRequests</span>) when <span style="background:none repeat scroll 0% 0% yellow" class="">NumRequests</span> > 0 -><br>
{<span style="background:none repeat scroll 0% 0% yellow" class="">ok</span>, {200, <<"OK">>}} = <span style="background:none repeat scroll 0% 0% yellow" class="">httpc</span>:request(get, {<span style="background:none repeat scroll 0% 0% yellow" class="">Url</span>, []}, [], [{body_format, binary}, {full_result, false}], <span style="background:none repeat scroll 0% 0% yellow" class="">ProfilePid</span>),<br>
<span style="background:none repeat scroll 0% 0% yellow" class="">io</span>:format("client ~p request completed.~n", [self()]),<br> <span style="background:none repeat scroll 0% 0% yellow" class="">io</span>:format("<span style="background:none repeat scroll 0% 0% yellow" class="">ets</span>:info(stand_alone_my_profile__handler_db, info) => ~p~n", [<span style="background:none repeat scroll 0% 0% yellow" class="">ets</span>:info(stand_alone_my_profile__handler_db, size)]),<br>
worker_thread(<span style="background:none repeat scroll 0% 0% yellow" class="">Url</span>, <span style="background:none repeat scroll 0% 0% yellow" class="">ProfilePid</span>, <span style="background:none repeat scroll 0% 0% yellow" class="">NumRequests</span> - 1).<br>
<br>%<span style="background:none repeat scroll 0% 0% yellow" class="">EOF</span><br><br>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>><br>
<br><br></div><div>Proposed patch:<br>------------------------<br><br>On file : <span style="background:none repeat scroll 0% 0% yellow" class="">otp</span>_<span style="background:none repeat scroll 0% 0% yellow" class="">src</span>_R16B03-1/lib/<span style="background:none repeat scroll 0% 0% yellow" class="">inets</span>/<span style="background:none repeat scroll 0% 0% yellow" class="">src</span>/http_client/<span style="background:none repeat scroll 0% 0% yellow" class="">httpc</span>_manager.<span style="background:none repeat scroll 0% 0% yellow" class="">erl</span> (didn't have the R17 src)<br>
<br>82a83<br>> Server = {local, <span style="background:none repeat scroll 0% 0% yellow" class="">ProfileName</span>}, <br>86c87<br>< gen_server:start_link(?MODULE, <span style="background:none repeat scroll 0% 0% yellow" class="">Args</span>, Opts);<br>
---<br>> gen_server:start_link(Server, ?MODULE, <span style="background:none repeat scroll 0% 0% yellow" class="">Args</span>, Opts);<br><br></div><div><br></div><div>Other notes:<br>------------------<br><br></div>
<div>The reason why this issue never surfaced in R14B04, is because of the<br></div><div><span style="background:none repeat scroll 0% 0% yellow" class="">kepos</span> specified when creating the __handler_db <span style="background:none repeat scroll 0% 0% yellow" class="">ETS</span> table :<br>
<br></div><div>In file <span style="background:none repeat scroll 0% 0% yellow" class="">otp</span>_<span style="background:none repeat scroll 0% 0% yellow" class="">src</span>_R14B04/lib/<span style="background:none repeat scroll 0% 0% yellow" class="">inets</span>/<span style="background:none repeat scroll 0% 0% yellow" class="">src</span>/http_client/<span style="background:none repeat scroll 0% 0% yellow" class="">httpc</span>_manager.<span style="background:none repeat scroll 0% 0% yellow" class="">erl</span>:<br>
</div><div><br></div><div>You have :<br>-record(handler_info,<br> {<br> id, % Id of the request: request_id()<br> starter, % <span style="background:none repeat scroll 0% 0% yellow" class="">Pid</span> of the handler starter process (temp): <span style="background:none repeat scroll 0% 0% yellow" class="">pid</span>()<br>
handler, % <span style="background:none repeat scroll 0% 0% yellow" class="">Pid</span> of the handler process: <span style="background:none repeat scroll 0% 0% yellow" class="">pid</span>()<br> from, % From for the request: from()<br>
state % State of the handler: initiating | started | operational | canceled<br> }).<br><br></div><div>and in do_<span style="background:none repeat scroll 0% 0% yellow" class="">init</span>/2 :<br><br><span style="background:none repeat scroll 0% 0% yellow" class="">ets</span>:new(<span style="background:none repeat scroll 0% 0% yellow" class="">HandlerDbName</span>,<br>
[protected, set, named_table, {<span style="background:none repeat scroll 0% 0% yellow" class="">keypos</span>, #<a href="http://handler_info.id">handler_info.id</a>}]),<br><br>#<a href="http://handler_info.id">handler_info.id</a> maps to element at position 2 in the tuple (element 1 is the record name),<br>
which means the <span style="background:none repeat scroll 0% 0% yellow" class="">httpc</span>_handler <span style="background:none repeat scroll 0% 0% yellow" class="">Pid</span> <br></div><div>is used as key for adding new entries to __handler_db, instead of the <br>
</div><div>reference generated for each new request (see lines 656 and 716 in same file).<br></div><div>The net effect being that the __handler_db table being "cleaned" by overriding any<br></div><div>existing request entry to a given <span style="background:none repeat scroll 0% 0% yellow" class="">httpc</span>_handler <span style="background:none repeat scroll 0% 0% yellow" class="">pid</span>.<br>
</div><div><br></div><div>In later versions (R15 and up), the <span style="background:none repeat scroll 0% 0% yellow" class="">ETS</span> table is created with :<br><span style="background:none repeat scroll 0% 0% yellow" class="">ets</span>:new(<span style="background:none repeat scroll 0% 0% yellow" class="">HandlerDbName</span>, [protected, set, named_table, {<span style="background:none repeat scroll 0% 0% yellow" class="">keypos</span>, 1}]) <br>
<br></div><div>which results in the reference associated with each request being used for<br></div><div>creating new entries in the relevant __handler_db table, causing the memory leak.<br></div><div><div><div><div><br></div>
<div>--END--<br></div><div><br>-- <br><span style="background:none repeat scroll 0% 0% yellow" class="">Ruan</span> <span style="background:none repeat scroll 0% 0% yellow" class="">Jonker</span><br>South Africa<br>+27824619036<br>
</div></div></div></div></div>