[erlang-questions] Profiling tool to find bottleneck on my Erlang code

[Knut Nesheim]

On Thu, Oct 6, 2011 at 8:30 PM, Zabrane Mickael <zabrane3@REDACTED> wrote:
> Hi guys,
>
> We have at work, a heavily parallel Erlang application running for years now.
> We would like to concentrate our efforts in the next couple of months in optimizing it bit more (if possible).
>
> Could someone please point me to a good "high level profiling tool" (not eprof/fprof which we already now,
> and find not very sexy) to tracks bottlenecks?

I have been working on a tool for our application to accomplish this.
It is still very much early stages, but it's already available on
github.com/knutin/kprof

The idea is to trace the path of execution of a request in your
system, as it travels through different tiers and different processes.
The runtime[1] of each tier is aggregated per request "id" and tier.
The request is identified by a function you provide, so you can for
example know the performance per API call, even though they may all
travel through the same tiers. For example: "list all invoices" spends
50ms in the database tier and 200ms rendering the pdf response .
"project future earnings" spends 500ms in the business logic, but
almost nothing in the database, etc.

The trace is collected and aggregated inside your node. The timings
can be stored in couchdb where you can view them using a couchapp that
creates nice graphs and tells you standard deviation, percentiles,
throughput, etc. Sending the trace messages to an external node is
also something I have looked into.

I want it to run continuously in live systems, tracing all requests
through a certain entry point or sampling them(sampling is not yet
implemented though).

I've been working on this project for use in our game servers, which
are high throughput/low latency so we care very much about the
slow-down introduced by tracing. As it is restricted to only a few
functions(typically one in each tier), it is not that critical. With
the sampling you can fine tune. Measuring the exact impact is on my
TODO list.

I would love some feedback on the idea and the project. If you have
any needs that is not solved by what I mentioned above please share so
I can understand what it would take to make the tool fit more than
just our use case. If anyone is interested in working together on the
project, this would also be really cool.

Thanks
Knut

[1]: And time spent in gc, if I get around to implementing it

>
> Any advice will be appreciated.
>
> Regards,
> Zabrane
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>