[erlang-questions] Re: Possible leak in inets httpc.

Anthony Molinaro anthonym@REDACTED
Wed Jul 14 22:26:30 CEST 2010


Well, the leak was there in a big way under production traffic patterns, so
I switched from httpc to ibrowse.  All the leaks went away.  Just an FYI
for anyone out there making a decision between httpc and ibrowse.

-Anthony

On Fri, Jul 09, 2010 at 09:26:48AM -0700, Anthony Molinaro wrote:
> Yeah, I managed to get the entire set of patches, basically used
> 
> git diff 201ccdd08bd2a591fe1e348a9a1df3d94a3606e4 141ff3c087ca856d21d0 lib/inets
> 
> Which seemed to get them all.  However, I still see memory leaked
> 
> Erlang R13B04 (erts-5.7.5) [source] [64-bit] [smp:2:2] [rq:2] [async-threads:0]
> [kernel-poll:false]
> 
> Eshell V5.7.5  (abort with ^G)
> 1> inets:start().
> ok
> 2> ets:i().
> ...
>  16402           httpc_manager__session_cookie_db bag   0      316      httpc_manager
>  httpc_manager__handler_db httpc_manager__handler_db set   0      316      httpc_manager
>  httpc_manager__session_db httpc_manager__session_db set   0      316      httpc_manager
> ...
> 3> http:request (post, { "http://www.example.com", [], "application/octet-stream", <<"hello">>}, [{connect_timeout, 100},{timeout, 10}], [{body_format, binary} ]).
> {error,timeout}
> 4> ets:i().
> ...
>  16402           httpc_manager__session_cookie_db bag   0      316      httpc_manager
>  httpc_manager__handler_db httpc_manager__handler_db set   1      333      httpc_manager
>  httpc_manager__session_db httpc_manager__session_db set   0      316      httpc_manager
> ...
> 5> ets:i(httpc_manager__handler_db).
> <1   > {handler_info,#Ref<0.0.0.142>,undefined,<0.60.0>,<0.33.0>,operational}
> 
> 6> m(inets).
> Module inets compiled: Date: July 8 2010, Time: 06.45
> Compiler options:  [{d,'SERVER_SOFTWARE',"inets/5.3.3"},
>                     {cwd,"/home/molinaro/projects/work/vendor/erlang/R13B04/rpmbuild/BUILD/otp_src_R13B04/lib/inets/src/inets_app"},
>                     {outdir,"/home/molinaro/projects/work/vendor/erlang/R13B04/rpmbuild/BUILD/otp_src_R13B04/lib/inets/src/inets_app/../../ebin"},
>                     {attribute,insert,app_vsn,"inets-5.3.3"},
>                     {parse_transform,sys_pre_attributes},
>                     debug_info]
> Object file: /usr/lib64/erlang/lib/inets-5.3.3/ebin/inets.beam
> ...
> 
> So it seems if you connect but then timeout you leak some memory even with
> 5.3.3.  I've not tried 5.4 which is in dev but the only changes there seem
> to be related to ssl...
> 
> Can anyone else confirm this is still broken?
> 
> -Anthony
> 
> On Fri, Jul 09, 2010 at 12:12:13PM +0200, Micael Karlberg wrote:
> > This problem was fixed in inets-5.3.2. The fix was reported by Lev Walkin:
> > 
> > http://www.erlang.org/cgi-bin/ezmlm-cgi/2/1766
> > 
> > Unfortunately, github was not updated in one commit, but in several, here
> > are some of them:
> > 
> > http://github.com/erlang/otp/commit/4b87b22ce97abf2759eb551222a862e17c5f4dcb
> > http://github.com/erlang/otp/commit/91c89d54d45989a85367f10d5902b9b508754a49
> > 
> > So you have to do a bit of cut-and-paste. The last version of inets in
> > R13B was 5.3.3,
> > but I don't know if that made it into github.
> > 
> > Regards,
> >     /BMK
> > 
> > On Fri, Jul 9, 2010 at 8:12 AM, Andrey Popp <8mayday@REDACTED> wrote:
> > > We have updated to R14 (thanks to deb packages in debian squeeze
> > > repository) and all seems to be ok for now.
> > >
> > > On Fri, Jul 9, 2010 at 1:34 AM, Anthony Molinaro
> > > <anthonym@REDACTED> wrote:
> > >> Well, looks like the answer is yes and no.  I was able to get a patch
> > >> which applied cleanly, however, the problem does not appear to be fixed.
> > >> The ets table still fills up with every request that times out
> > >> (I unfortunately don't have a standalone test, but I've tested
> > >> repeatedly and it always seem to get an extra entry for every request
> > >> that times out).
> > >>
> > >> This is sort of a show stopper for me upgrading, so I'll be trying to
> > >> figure out a fix, but if anyone more familiar with the code wants to
> > >> help me out, it would be appreciated.
> > >>
> > >> Thanks,
> > >>
> > >> -Anthony
> > >>
> > >> On Thu, Jul 08, 2010 at 01:13:41PM -0700, Anthony Molinaro wrote:
> > >>> Hi, anyone know if its safe to backport this fix to R13B04?  I'm in the process
> > >>> of deploying R13B04 as an upgrade from R12B05 and since I make heavy use of
> > >>> httpc this will bite me.  Ideally, I can just grab the appropriate files from
> > >>> git and plop them into my R13B04 when I build an rpm (of course I need to
> > >>> remember enough git to do that :) ), so wondering if it's possible or if
> > >>> there are deps outside of the inets directory which matter?
> > >>>
> > >>> Thanks,
> > >>>
> > >>> -Anthony
> > >>>
> > >>> On Thu, Jul 08, 2010 at 02:12:07PM +0400, Andrey Popp wrote:
> > >>> > It seems it has been fixed with [1] in inets-5.3.2. Sorry.
> > >>> >
> > >>> > [1]: http://github.com/erlang/otp/commit/91c89d54d45989a85367f10d5902b9b508754a49
> > >>> >
> > >>> > On Thu, Jul 8, 2010 at 1:57 PM, Andrey Popp <8mayday@REDACTED> wrote:
> > >>> > > Hello,
> > >>> > >
> > >>> > > I have application that makes huge number of HTTP requests with httpc
> > >>> > > and I have spotted that `httpc_manager__handler_db` ETS table growing
> > >>> > > in size and doesn't clean up. That is the example records from this
> > >>> > > table (ets:i(httpc_manager__handler_db)):
> > >>> > >
> > >>> > > <1   > {handler_info,#Ref<0.0.0.18800>,undefined,<0.177.0>,undefined,operational}
> > >>> > > <2   > {handler_info,#Ref<0.0.0.35242>,undefined,<0.3075.0>,undefined,operational}
> > >>> > > <3   > {handler_info,#Ref<0.0.0.61713>,undefined,<0.5755.0>,undefined,operational}
> > >>> > > <4   > {handler_info,#Ref<0.0.0.68114>,undefined,<0.6943.0>,undefined,operational}
> > >>> > > <5   > {handler_info,#Ref<0.0.0.85303>,undefined,<0.8305.0>,undefined,operational}
> > >>> > >
> > >>> > > and processes with handler pids are no longer alive. Also, note that I
> > >>> > > have no timeout setting in my http:request HTTPOptions, but I think
> > >>> > > inets should not leak even in that case.
> > >>> > >
> > >>> > > Thanks.
> > >>> > >
> > >>> > > --
> > >>> > > Andrey Popp
> > >>> > >
> > >>> > > phone: +7 911 740 24 91
> > >>> > > e-mail: 8mayday@REDACTED
> > >>> > >
> > >>> >
> > >>> >
> > >>> >
> > >>> > --
> > >>> > Andrey Popp
> > >>> >
> > >>> > phone: +7 911 740 24 91
> > >>> > e-mail: 8mayday@REDACTED
> > >>> >
> > >>> > ________________________________________________________________
> > >>> > erlang-questions (at) erlang.org mailing list.
> > >>> > See http://www.erlang.org/faq.html
> > >>> > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
> > >>> >
> > >>>
> > >>> --
> > >>> ------------------------------------------------------------------------
> > >>> Anthony Molinaro                           <anthonym@REDACTED>
> > >>
> > >> --
> > >> ------------------------------------------------------------------------
> > >> Anthony Molinaro                           <anthonym@REDACTED>
> > >>
> > >> ________________________________________________________________
> > >> erlang-questions (at) erlang.org mailing list.
> > >> See http://www.erlang.org/faq.html
> > >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
> > >>
> > >>
> > >
> > >
> > >
> > > --
> > > Andrey Popp
> > >
> > > phone: +7 911 740 24 91
> > > e-mail: 8mayday@REDACTED
> > >
> > > ________________________________________________________________
> > > erlang-questions (at) erlang.org mailing list.
> > > See http://www.erlang.org/faq.html
> > > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
> > >
> > >
> > 
> > ________________________________________________________________
> > erlang-questions (at) erlang.org mailing list.
> > See http://www.erlang.org/faq.html
> > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
> > 
> 
> -- 
> ------------------------------------------------------------------------
> Anthony Molinaro                           <anthonym@REDACTED>

-- 
------------------------------------------------------------------------
Anthony Molinaro                           <anthonym@REDACTED>


More information about the erlang-questions mailing list