[erlang-questions] widefinder update

[Anders Nygren]

On 11/1/07, Thomas Lindgren <thomasl_erlang@REDACTED> wrote:
>
> --- Anders Nygren <anders.nygren@REDACTED> wrote:
>
> > On 10/29/07, Thomas Lindgren
> > <thomasl_erlang@REDACTED> wrote:
> > >
> > > --- Steve Vinoski <vinoski@REDACTED> wrote:
> > >
> > > > On 10/28/07, Anders Nygren
> > <anders.nygren@REDACTED>
> > > > wrote:
> > > > >
> > > > > On 10/28/07, Hynek Vychodil
> > > > <vychodil.hynek@REDACTED> wrote:
> > > > > > Hi Anders,
> > > > > > I rewrote your code a little. I removed all
> > > > remaining binary bindings
> > > > > > and it is noticeable faster again. Try
> > > > wf_pichi3.erl.
> > > > > >
> > > > >
> > > > > Hynek
> > > > > that was great, Your change brings my
> > wfinder1_1 +
> > > > wfbm4_ets1_1
> > > > > down to
> > > > > real    0m1.118s
> > > > > user    0m1.640s
> > > > > sys     0m0.368s
> > > > > on my 1.66 GHz dual core laptop.
> > > >
> > > >
> > > > And on the 8-core 2.33GHz Intel Xeon Linux box
> > with
> > > > 2 GB RAM, this version
> > > > is extremely fast:
> > > >
> > > > real 0m0.567s
> > > > user 0m2.249s
> > > > sys 0m0.956s
> > >
> > > (I'll ignore the unexplained sys time below. That
> > > makes the discussion a bit preliminary; perhaps
> > the
> > > derived results should be computed some other way.
> > > Apply grain of salt appropriately.)
> > >
> > > For those keeping track, the latest result is
> > fully
> > > 2.7 times faster than the best previous version
> > (which
> > > was block read), and 17.3 times faster than the
> > > initial version. The latest speedup is basically
> > due
> > > to doing less work.
> >
> > Are we doing less work?
> > Not really, over 3 versions wfinder, wfinder1 and
> > wfinder1_1
> > the only differences are how we do binary matching
> > - do NOT create sub binaries
> > - keep "pointers" for offsets inside one big binary
> > The reason that it get so much faster seems to be
> > that
> > we have much less garbage collection.
>
> Just to be clear, here I put memory allocation and
> reclamation in the category of 'work'. So doing less
> work in this sense means being more efficient for the
> same data set. In the case of widefinder, the user
> time has steadily decreased with new versions, which I
> take to indicate precisely this.
>

Yes, I understood that, but my point was that we had
not decreased the amount of work by any fundamentally
different algorithms.

/Anders