[erlang-questions] Improving performance on a thrift to protobuf proxy

[Anthony Molinaro]

Hi,

  While I haven't yet answered all my own questions I did discover 2 things
which might help future erlang programmers.

1. use ibrowse as an http client, not httpc.  httpc leaks memory and is buggy
   in R13B04
2. R13B04 is better than R12B05, much better.  It used half the CPU (or at
   least did after switching from httpc to ibrowse).

If I discover more I will send more.

-Anthony

On Mon, Jun 14, 2010 at 11:52:59AM -0700, Anthony Molinaro wrote:
> Hi,
> 
>   I have a thrift server (using the generated thrift server from the thrift
> distribution) which contains a single method.  The method takes two thrift
> objects (a context, and a list of places to call).  The thrift method handler
> spawns a process for each place to call.  Each of those processes, takes the
> context, combines it with some meta info particular to the place to call,
> and serializes the request as a protobuf.  It then makes an http call using
> httpc with a timeout, and calls the place to call passing the protobuf as the
> body of an HTTP GET request.
> 
>   For each incoming request I end up with 6-10 outgoing requests.  Right now
> the server runs pretty well, but it seems to use a lot of CPU, also even
> though I set a pretty tight timeout on my httpc call (120 msecs), the
> server often doesn't respond within 200 msecs to the caller.
> 
>   I did a little bit of profiling in isolation (using eprof), but that didn't
> really show any low hanging fruit (however, I'm not entirely sure I used the
> right profiling).  So I'm looking for pointers to help improve throughput
> and lower CPU usage.
> 
> Specifically,
> 
>   - what is the best way to profile this sort of application?  using tracing?
>     fprof? eprof? on the live server or try to test in isolation (I profiled
>     in isolation before, but as I said no real low hanging fruit).
>   - is httpc the ideal http client?  are there better http client libraries?
>     should I roll my own using event driven non-blocking techniques?  does
>     the timeout in httpc really work, or should I do something with a timer?
>   - this is really just sort of a glorified web crawler, has anyone out there
>     written a crawler and have pointers?
> 
> Any other hints or tips would be greatly appreciated.
> 
> Thanks,
> 
> -Anthony
> 
> -- 
> ------------------------------------------------------------------------
> Anthony Molinaro                           <anthonym@REDACTED>
> 
> ________________________________________________________________
> erlang-questions (at) erlang.org mailing list.
> See http://www.erlang.org/faq.html
> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
> 

-- 
------------------------------------------------------------------------
Anthony Molinaro                           <anthonym@REDACTED>