profiling network apps: fprof vs sequential tracing

[Joel Reymont]

Suppose I'm trying to profile a web app running on top of mochiweb. 

The app starts processes for every request coming in and these talk to other processes that use the disk, etc. There are a few hundred requests in a single benchmarking run. 

I'm measuring disk write performance and see that it slows down to a crawl sometimes. 

Has anyone used fprof or sequential tracing to identify bottlenecks in this or similar scenario?

The problem with fprof is that it aggregates by process. This means I have several hundred profiling sets, one per request, in a 79mb text file. It's very hard to find a bottleneck in such a huge file, specially when it's split per process. 

I think I'll need to further massage this file to establish a path a new request goes through, from the time it hits the server to the time output is sent. I could then analyze several hundred paths like this and look for statistical outliers. 

I'm wondering if it's better to use sequential tracing in this scenario. Creating a new token when the request hits the app should automatically establish the path I'm looking for and allow me to time each segment. 

Am I on the right track? Has anyone dealt with a similar situation?

	Thanks, Joel

---
http://twitter.com/wagerlabs