[erlang-questions] EEP0018/native JSON parsing

[Jan Lehnardt]

On 26 Jan 2009, at 01:24, Paul Davis wrote:

> After a bit of pestering from Jan, I broke down and tested the two
> versions side by side. The code for both versions are in my github
> repo. Also, disregard the errors on eno's test runs. He's using a
> different number format than is expected so nearly all the tests fail.

The number tests fail since Enrico maintains bignum precision on the
C-end. Which none of the existing JSON libs do, but is rather neat,
but you (the Erlang experts) would know if we need that.

Cheers
Jan
--
> Anyway, numbers:
>
> davis2318:/usr/local/src/eep0018 davisp$ git branch
>  eno
> * master
>  pjd
>  sax
> davis2318:/usr/local/src/eep0018 davisp$ make check
> for d in src tests; do (cd $d; make); done
>
>    [snip]
>
> for d in src tests; do (cd $d; make check); done
> cp ../src/*.so ./
> cp ../src/*.beam ./
> erl -noshell -s runner main
> single_proc =>    eep0018:  0.417 s [success]
> multi_proc =>    eep0018:  1.396 s [success]
> single_proc => mochijson2:  3.191 s [success]
> multi_proc => mochijson2:  9.915 s [success]
> single_proc =>   rabbitmq:  2.904 s [success]
> multi_proc =>   rabbitmq:  9.510 s [success]
> davis2318:/usr/local/src/eep0018 davisp$ git checkout eno
> Switched to branch "eno"
> davis2318:/usr/local/src/eep0018 davisp$ make clean && make check
> for d in src tests; do (cd $d; make clean); done
> rm -f *.o *.beam eep0018_drv.so
> rm -f *.so *.beam
> for d in src tests; do (cd $d; make); done
>
>    [snip]
>
> for d in src tests; do (cd $d; make check); done
> cp ../src/*.so ./
> cp ../src/*.beam ./
> erl -noshell -s runner main
> single_proc =>    eep0018:  0.698 s [failure]
>        19 decoding errors.
>        19 round trip errors.
> multi_proc =>    eep0018:  3.197 s [failure]
>        95 decoding errors.
>        95 round trip errors.
> single_proc => mochijson2:  3.161 s [success]
> multi_proc => mochijson2:  9.895 s [success]
> single_proc =>   rabbitmq:  3.008 s [success]
> multi_proc =>   rabbitmq: 10.090 s [success]
>
> On Sun, Jan 25, 2009 at 4:49 PM, Enrico Thierbach
> <enrico.thierbach@REDACTED> wrote:
>> Hi guys,
>>
>> I have just finished what I would call the first stage of the native
>> JSON parser implementation. This is the state as of now at
>> http://github.com/pboy/eep0018/, Please see the readme file.
>>
>> In short, this is the status:
>>
>> - I parse everything that comes along like mochijson2 and rabbitmq
>> - optionally I can parse according to eep0018
>> - my code runs 6 times as fast as  mochijson2/rabbitmq at JSON input
>> of a certain size, and is usally not slower on very small JSON input.
>>
>> jan tried the module along with couchdb, and find one issue regarding
>> UTF8 characters; besides of that everything seemed to run fine and
>> much faster. The utf8 parsing issue is resolved (or better: worked
>> around: the JSON parser and the CouchDB tests have different ideas on
>> what is valid UTF8).
>>
>> What would be next?
>>
>> 1. I would like to invite you all to review and try the code.
>>
>> 2. I need some hints regarding "parallel execution". The native  
>> driver
>> does not support multithreading (and why should it? It only
>> complicates things where OTP can do that kind of stuff by itself
>> already.) With the current code the driver gets loaded only once.
>> Therefore on a multicore machine only one CPU gets really used. Is it
>> somehow possible to load the driver multiple times? The only way I  
>> see
>> so far is having the driver compiled and installed multiple times  
>> with
>> different names; but I guess there is a better way. The code itself
>> should luckily run in a parallel situation.
>>
>> 3. and finally I'll have to tackle the Erlang->JSON issue. I don't
>> expect a speedup as big.
>>
>> Please see my next mail for some comments on EEP0018.
>>
>> /eno
>>
>> ====================================================================
>> A wee piece of ruby every monday: http://1rad.wordpress.com/
>>
>