[erlang-questions] Re: Possible leak in inets httpc.
Anthony Molinaro
anthonym@REDACTED
Wed Jul 14 22:26:30 CEST 2010
Well, the leak was there in a big way under production traffic patterns, so
I switched from httpc to ibrowse. All the leaks went away. Just an FYI
for anyone out there making a decision between httpc and ibrowse.
-Anthony
On Fri, Jul 09, 2010 at 09:26:48AM -0700, Anthony Molinaro wrote:
> Yeah, I managed to get the entire set of patches, basically used
>
> git diff 201ccdd08bd2a591fe1e348a9a1df3d94a3606e4 141ff3c087ca856d21d0 lib/inets
>
> Which seemed to get them all. However, I still see memory leaked
>
> Erlang R13B04 (erts-5.7.5) [source] [64-bit] [smp:2:2] [rq:2] [async-threads:0]
> [kernel-poll:false]
>
> Eshell V5.7.5 (abort with ^G)
> 1> inets:start().
> ok
> 2> ets:i().
> ...
> 16402 httpc_manager__session_cookie_db bag 0 316 httpc_manager
> httpc_manager__handler_db httpc_manager__handler_db set 0 316 httpc_manager
> httpc_manager__session_db httpc_manager__session_db set 0 316 httpc_manager
> ...
> 3> http:request (post, { "http://www.example.com", [], "application/octet-stream", <<"hello">>}, [{connect_timeout, 100},{timeout, 10}], [{body_format, binary} ]).
> {error,timeout}
> 4> ets:i().
> ...
> 16402 httpc_manager__session_cookie_db bag 0 316 httpc_manager
> httpc_manager__handler_db httpc_manager__handler_db set 1 333 httpc_manager
> httpc_manager__session_db httpc_manager__session_db set 0 316 httpc_manager
> ...
> 5> ets:i(httpc_manager__handler_db).
> <1 > {handler_info,#Ref<0.0.0.142>,undefined,<0.60.0>,<0.33.0>,operational}
>
> 6> m(inets).
> Module inets compiled: Date: July 8 2010, Time: 06.45
> Compiler options: [{d,'SERVER_SOFTWARE',"inets/5.3.3"},
> {cwd,"/home/molinaro/projects/work/vendor/erlang/R13B04/rpmbuild/BUILD/otp_src_R13B04/lib/inets/src/inets_app"},
> {outdir,"/home/molinaro/projects/work/vendor/erlang/R13B04/rpmbuild/BUILD/otp_src_R13B04/lib/inets/src/inets_app/../../ebin"},
> {attribute,insert,app_vsn,"inets-5.3.3"},
> {parse_transform,sys_pre_attributes},
> debug_info]
> Object file: /usr/lib64/erlang/lib/inets-5.3.3/ebin/inets.beam
> ...
>
> So it seems if you connect but then timeout you leak some memory even with
> 5.3.3. I've not tried 5.4 which is in dev but the only changes there seem
> to be related to ssl...
>
> Can anyone else confirm this is still broken?
>
> -Anthony
>
> On Fri, Jul 09, 2010 at 12:12:13PM +0200, Micael Karlberg wrote:
> > This problem was fixed in inets-5.3.2. The fix was reported by Lev Walkin:
> >
> > http://www.erlang.org/cgi-bin/ezmlm-cgi/2/1766
> >
> > Unfortunately, github was not updated in one commit, but in several, here
> > are some of them:
> >
> > http://github.com/erlang/otp/commit/4b87b22ce97abf2759eb551222a862e17c5f4dcb
> > http://github.com/erlang/otp/commit/91c89d54d45989a85367f10d5902b9b508754a49
> >
> > So you have to do a bit of cut-and-paste. The last version of inets in
> > R13B was 5.3.3,
> > but I don't know if that made it into github.
> >
> > Regards,
> > /BMK
> >
> > On Fri, Jul 9, 2010 at 8:12 AM, Andrey Popp <8mayday@REDACTED> wrote:
> > > We have updated to R14 (thanks to deb packages in debian squeeze
> > > repository) and all seems to be ok for now.
> > >
> > > On Fri, Jul 9, 2010 at 1:34 AM, Anthony Molinaro
> > > <anthonym@REDACTED> wrote:
> > >> Well, looks like the answer is yes and no. I was able to get a patch
> > >> which applied cleanly, however, the problem does not appear to be fixed.
> > >> The ets table still fills up with every request that times out
> > >> (I unfortunately don't have a standalone test, but I've tested
> > >> repeatedly and it always seem to get an extra entry for every request
> > >> that times out).
> > >>
> > >> This is sort of a show stopper for me upgrading, so I'll be trying to
> > >> figure out a fix, but if anyone more familiar with the code wants to
> > >> help me out, it would be appreciated.
> > >>
> > >> Thanks,
> > >>
> > >> -Anthony
> > >>
> > >> On Thu, Jul 08, 2010 at 01:13:41PM -0700, Anthony Molinaro wrote:
> > >>> Hi, anyone know if its safe to backport this fix to R13B04? I'm in the process
> > >>> of deploying R13B04 as an upgrade from R12B05 and since I make heavy use of
> > >>> httpc this will bite me. Ideally, I can just grab the appropriate files from
> > >>> git and plop them into my R13B04 when I build an rpm (of course I need to
> > >>> remember enough git to do that :) ), so wondering if it's possible or if
> > >>> there are deps outside of the inets directory which matter?
> > >>>
> > >>> Thanks,
> > >>>
> > >>> -Anthony
> > >>>
> > >>> On Thu, Jul 08, 2010 at 02:12:07PM +0400, Andrey Popp wrote:
> > >>> > It seems it has been fixed with [1] in inets-5.3.2. Sorry.
> > >>> >
> > >>> > [1]: http://github.com/erlang/otp/commit/91c89d54d45989a85367f10d5902b9b508754a49
> > >>> >
> > >>> > On Thu, Jul 8, 2010 at 1:57 PM, Andrey Popp <8mayday@REDACTED> wrote:
> > >>> > > Hello,
> > >>> > >
> > >>> > > I have application that makes huge number of HTTP requests with httpc
> > >>> > > and I have spotted that `httpc_manager__handler_db` ETS table growing
> > >>> > > in size and doesn't clean up. That is the example records from this
> > >>> > > table (ets:i(httpc_manager__handler_db)):
> > >>> > >
> > >>> > > <1 > {handler_info,#Ref<0.0.0.18800>,undefined,<0.177.0>,undefined,operational}
> > >>> > > <2 > {handler_info,#Ref<0.0.0.35242>,undefined,<0.3075.0>,undefined,operational}
> > >>> > > <3 > {handler_info,#Ref<0.0.0.61713>,undefined,<0.5755.0>,undefined,operational}
> > >>> > > <4 > {handler_info,#Ref<0.0.0.68114>,undefined,<0.6943.0>,undefined,operational}
> > >>> > > <5 > {handler_info,#Ref<0.0.0.85303>,undefined,<0.8305.0>,undefined,operational}
> > >>> > >
> > >>> > > and processes with handler pids are no longer alive. Also, note that I
> > >>> > > have no timeout setting in my http:request HTTPOptions, but I think
> > >>> > > inets should not leak even in that case.
> > >>> > >
> > >>> > > Thanks.
> > >>> > >
> > >>> > > --
> > >>> > > Andrey Popp
> > >>> > >
> > >>> > > phone: +7 911 740 24 91
> > >>> > > e-mail: 8mayday@REDACTED
> > >>> > >
> > >>> >
> > >>> >
> > >>> >
> > >>> > --
> > >>> > Andrey Popp
> > >>> >
> > >>> > phone: +7 911 740 24 91
> > >>> > e-mail: 8mayday@REDACTED
> > >>> >
> > >>> > ________________________________________________________________
> > >>> > erlang-questions (at) erlang.org mailing list.
> > >>> > See http://www.erlang.org/faq.html
> > >>> > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
> > >>> >
> > >>>
> > >>> --
> > >>> ------------------------------------------------------------------------
> > >>> Anthony Molinaro <anthonym@REDACTED>
> > >>
> > >> --
> > >> ------------------------------------------------------------------------
> > >> Anthony Molinaro <anthonym@REDACTED>
> > >>
> > >> ________________________________________________________________
> > >> erlang-questions (at) erlang.org mailing list.
> > >> See http://www.erlang.org/faq.html
> > >> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
> > >>
> > >>
> > >
> > >
> > >
> > > --
> > > Andrey Popp
> > >
> > > phone: +7 911 740 24 91
> > > e-mail: 8mayday@REDACTED
> > >
> > > ________________________________________________________________
> > > erlang-questions (at) erlang.org mailing list.
> > > See http://www.erlang.org/faq.html
> > > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
> > >
> > >
> >
> > ________________________________________________________________
> > erlang-questions (at) erlang.org mailing list.
> > See http://www.erlang.org/faq.html
> > To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
> >
>
> --
> ------------------------------------------------------------------------
> Anthony Molinaro <anthonym@REDACTED>
--
------------------------------------------------------------------------
Anthony Molinaro <anthonym@REDACTED>
More information about the erlang-questions
mailing list