[erlang-questions] io_lib and unicode post processing?

[Musumeci, Antonio S]

It's not the generation of the JSON. I've tested that and it makes up only a small portion of the overhead. It's most of it is after I hand it off to webmachine. Even if I create a large binary and pass it along it does the same.

-----Original Message-----
From: Tim Watson [mailto:watson.timothy@REDACTED] 
Sent: Monday, August 23, 2010 2:31 PM
To: Musumeci, Antonio S (Enterprise Infrastructure)
Cc: erlang-questions@REDACTED
Subject: Re: [erlang-questions] io_lib and unicode post processing?

On 23 Aug 2010, at 18:37, Musumeci, Antonio S wrote:

> I've got a webmachine/mochiweb setup and it seems much slower/higher CPU than I expected. I'm creating JSON for 10k records using jsonerl and sending on to webmachine. After the serializing of the data the CPU is still pegged and I see in etop io_lib and unicode being called. Does anyone know why those would be called? It takes several seconds.

Surely they're getting called a lot during serialisation. Haven't used jsonerl before but what kind of output is getting produced - strings, iolists, binaries? Does it not offer a streaming API so you can forward the data as multipart/chunked? There are some useful profiling tools available with OTP and there's a good library that'll allow you use them on production systems without incurring dangerous overhead - eper. You'll find it on google code IIRC. I've been using a newer json library called jsx recently (http://github.com/talentdeficit/jsx) and find it performs really well, although I haven't load tested it so don't know if it's production worthy.