process leak in http client?

Cliff Moon cliff@REDACTED
Tue Aug 25 20:03:24 CEST 2009


It looks like there's a process leak somewhere in Erlang's standard http 
client.  I have a scraping system which goes against a third party 
website.  Usually the scrapes work just fine, however every once in a 
while it bumps up against some rate limiting mechanism on the remote 
server and the server abruptly hangs up the tcp connection.

After a while the process list is filled up with procs that look like this:

<0.27047.5>           erlang:apply/2                         987     
1898    0
                      http:handle_answer/3                     
9             
<0.27094.3>           erlang:apply/2                         987     
1911    0
                      http:handle_answer/3                     
9             
<0.27114.3>           erlang:apply/2                         987     
1911    0
                      http:handle_answer/3                     
9             
<0.27121.5>           erlang:apply/2                         987     
1911    0
                      http:handle_answer/3                     
9             
<0.27137.3>           erlang:apply/2                         987     
1898    0
                      http:handle_answer/3                     
9             
<0.27153.1>           erlang:apply/2                         987     
1911    0
                      http:handle_answer/3                     
9             
<0.27159.3>           erlang:apply/2                         987     
1890    0
                      http:handle_answer/3                     
9             
<0.27202.5>           erlang:apply/2                         987     
1911    0
                      http:handle_answer/3                     
9             
<0.27223.1>           erlang:apply/2                         987     
1897    0
                      http:handle_answer/3                     
9             

Has anyone seen this behavior before?  Is it due to the occasional 
hangups that happen from the server?


More information about the erlang-questions mailing list