[erlang-questions] io_lib and unicode post processing?

[Musumeci, Antonio S]

The API for webmachine expects a callback for a particular content type. What I mean is that my function is called and I return a list or binary of the data and then it's no longer in my control. Whatever is happening is behind the scenes.

-----Original Message-----
From: Tim Watson [mailto:watson.timothy@REDACTED] 
Sent: Monday, August 23, 2010 2:53 PM
To: Musumeci, Antonio S (Enterprise Infrastructure)
Cc: erlang-questions@REDACTED
Subject: Re: [erlang-questions] io_lib and unicode post processing?

How do you "pass it on" to webmachine - REST call over HTTP, rpc (distributed Erlang), raw socket call? 

On 23 Aug 2010, at 19:46, Musumeci, Antonio S wrote:

> It's not the generation of the JSON. I've tested that and it makes up only a small portion of the overhead. It's most of it is after I hand it off to webmachine. Even if I create a large binary and pass it along it does the same.
> 
> -----Original Message-----
> From: Tim Watson [mailto:watson.timothy@REDACTED]
> Sent: Monday, August 23, 2010 2:31 PM
> To: Musumeci, Antonio S (Enterprise Infrastructure)
> Cc: erlang-questions@REDACTED
> Subject: Re: [erlang-questions] io_lib and unicode post processing?
> 
> On 23 Aug 2010, at 18:37, Musumeci, Antonio S wrote:
> 
>> I've got a webmachine/mochiweb setup and it seems much slower/higher CPU than I expected. I'm creating JSON for 10k records using jsonerl and sending on to webmachine. After the serializing of the data the CPU is still pegged and I see in etop io_lib and unicode being called. Does anyone know why those would be called? It takes several seconds.
> 
> Surely they're getting called a lot during serialisation. Haven't used jsonerl before but what kind of output is getting produced - strings, iolists, binaries? Does it not offer a streaming API so you can forward the data as multipart/chunked? There are some useful profiling tools available with OTP and there's a good library that'll allow you use them on production systems without incurring dangerous overhead - eper. You'll find it on google code IIRC. I've been using a newer json library called jsx recently (http://github.com/talentdeficit/jsx) and find it performs really well, although I haven't load tested it so don't know if it's production worthy.