Improving performance on a thrift to protobuf proxy

Anthony Molinaro anthonym@REDACTED
Mon Jun 14 20:52:59 CEST 2010


Hi,

  I have a thrift server (using the generated thrift server from the thrift
distribution) which contains a single method.  The method takes two thrift
objects (a context, and a list of places to call).  The thrift method handler
spawns a process for each place to call.  Each of those processes, takes the
context, combines it with some meta info particular to the place to call,
and serializes the request as a protobuf.  It then makes an http call using
httpc with a timeout, and calls the place to call passing the protobuf as the
body of an HTTP GET request.

  For each incoming request I end up with 6-10 outgoing requests.  Right now
the server runs pretty well, but it seems to use a lot of CPU, also even
though I set a pretty tight timeout on my httpc call (120 msecs), the
server often doesn't respond within 200 msecs to the caller.

  I did a little bit of profiling in isolation (using eprof), but that didn't
really show any low hanging fruit (however, I'm not entirely sure I used the
right profiling).  So I'm looking for pointers to help improve throughput
and lower CPU usage.

Specifically,

  - what is the best way to profile this sort of application?  using tracing?
    fprof? eprof? on the live server or try to test in isolation (I profiled
    in isolation before, but as I said no real low hanging fruit).
  - is httpc the ideal http client?  are there better http client libraries?
    should I roll my own using event driven non-blocking techniques?  does
    the timeout in httpc really work, or should I do something with a timer?
  - this is really just sort of a glorified web crawler, has anyone out there
    written a crawler and have pointers?

Any other hints or tips would be greatly appreciated.

Thanks,

-Anthony

-- 
------------------------------------------------------------------------
Anthony Molinaro                           <anthonym@REDACTED>


More information about the erlang-questions mailing list