[erlang-questions] Strange optimization result

[Jay Nelson]

Thomas Lindgren wrote:

 > 4. Perhaps a streaming interface would be even better?
 > Jay Nelson suggested one a few years ago.

I've been following this thread here and in the blogosphere, but I am  
very busy with some code I am working on and am trying valiantly to  
wait until I am free before giving an analysis.  It will take some  
time to discuss all the issues.

For those who haven't read the streaming paper it is on my website at  
http://www.duomark.com/erlang/index.html and describes how to build  
BIFs that can speed up binary handling.  That work was an initial  
study of whether binary handling could be accelerated by including a  
low-level library of binary functions.  It gives performance  
comparisons on an old VM for binary matching versus BIFs.  This  
problem is very amenable to some of the techniques presented in the  
paper -- in fact, similar types of problems were what I had in mind  
when I started the investigation.

I did further work related to the parsing of binaries over the last  
year, but it is very incomplete.  It will be 6 months or so (based on  
my past track record rather than my optimism) before I will have a  
more complete package and analysis of performance.  I expect to use  
Tim's problem as a motivation for the work.

The main issues I see are as follows:

1) erlang has lots of options, so the developers were able to try  
several approaches
2) The performance issues are non-obvious without experience
3) There has been a lot of code slinging and micro-tweaking with good  
progress, but not the real analysis of the problem that would  
identify the architectural underpinnings for a good solution  
(although the participants now have some good experience with  
different erlang tools)
4) Doing a proper account of #3 as Thomas suggested would give an  
indication of some potential for improving the erlang out-of-the-box  
performance on data analysis applications
5) The solutions are large and complicated pieces of code to replace  
a 5-liner; erlang should be able to do something useful in 10-20  
lines of code
6) While the participants have tried to wring every ounce of  
performance out of their solutions, the prospect of a general  
solution that could easily be adapted to other problems is strong and  
should be faster than a ruby implementation on multi-core, although  
probably a little slower than the special purpose solutions being  
offered

jay