[erlang-questions] io_lib and unicode post processing?
Musumeci, Antonio S
Mon Aug 23 20:46:52 CEST 2010
It's not the generation of the JSON. I've tested that and it makes up only a small portion of the overhead. It's most of it is after I hand it off to webmachine. Even if I create a large binary and pass it along it does the same.
From: Tim Watson [mailto:]
Sent: Monday, August 23, 2010 2:31 PM
To: Musumeci, Antonio S (Enterprise Infrastructure)
Subject: Re: [erlang-questions] io_lib and unicode post processing?
On 23 Aug 2010, at 18:37, Musumeci, Antonio S wrote:
> I've got a webmachine/mochiweb setup and it seems much slower/higher CPU than I expected. I'm creating JSON for 10k records using jsonerl and sending on to webmachine. After the serializing of the data the CPU is still pegged and I see in etop io_lib and unicode being called. Does anyone know why those would be called? It takes several seconds.
Surely they're getting called a lot during serialisation. Haven't used jsonerl before but what kind of output is getting produced - strings, iolists, binaries? Does it not offer a streaming API so you can forward the data as multipart/chunked? There are some useful profiling tools available with OTP and there's a good library that'll allow you use them on production systems without incurring dangerous overhead - eper. You'll find it on google code IIRC. I've been using a newer json library called jsx recently (http://github.com/talentdeficit/jsx) and find it performs really well, although I haven't load tested it so don't know if it's production worthy.
NOTICE: If you have received this communication in error, please destroy all electronic and paper copies and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers. If you cannot access these links, please notify us by reply message and we will send the contents to you. By messaging with Morgan Stanley you consent to the foregoing.
More information about the erlang-questions