[erlang-questions] widefinder update

Steve Vinoski <>
Tue Oct 23 22:41:58 CEST 2007

On 10/23/07, Anders Nygren <> wrote:
> To summarize my progress on the widefinder problem
> A few days ago I started with Steve Vinoski's tbray16.erl
> As a baseline on my 1.66 GHz dual core Centrino
> laptop, Linux,
> tbray16
> real    0m7.067s
> user    0m12.377s
> sys     0m0.584s

Anders, thanks for collecting and posting these. I've just performed a set
of new timings for all of them, as listed below. For each, I just ran this

time erl -smp -noshell -run <test_case> main o1000k.ap >/dev/null

where "<test_case>" is the name of the tbray test case file. All were
looped ten times, and I took the best timing for each. All tests were done
on my 8-core 2.33 GHz dual Intel Xeon with 2 GB RAM Linux box, in a local
(non-NFS) directory.

My original tbray16 runs in

real    0m3.162s
user    0m16.513s
sys     0m1.762s

I removed the dict used for the shift table,
> and changed the min_heap_size.
> That gave
> real    0m2.713s
> user    0m4.168s
> sys     0m0.412s
> (see tbray_tuple.erl and wfbm4_tuple.erl)
> Steve reported that it ran in ~1.9 s on his 8 core server.

What I get for tbray_tuple is:

real    0m2.285s
user    0m8.615s
sys     0m0.988s

> Then I removed the dicts that were used for collecting the
> matches and used ets instead, and got some improvement
> on my dual core laptop.
> real    0m2.220s
> user    0m3.252s
> sys     0m0.344s
> (see tbray_ets.erl and wfbm4_ets.erl)
> Interestingly Steve reported that it actually performed
> worse on his 8 core server.

The discrepancy seems to be gone. With your new file that you supplied in
your message, the official timing for tbray_ets on the 8-core is:

real    0m1.868s
user    0m7.416s
sys     0m0.509s

> These versions all read the whole file into memory at the start.
> On my laptop that takes ~400ms (when the file is already cached
> in the OS).
> So I changed it to read the file in chucks and spawn the worker
> after each chunk is read.
> tbray_blockread with 4 processes
> real    0m1.992s
> user    0m3.176s
> sys     0m0.420s
> (see tbray_blockread.erl and wfbm4_ets.erl)
> Running it in the erlang shell it takes ~1.8s.

Interestingly, some of my earlier attempts tried to overlap block reads and
worker spawning, but the results were always worse, so that's why I went to
reading in the whole file. This blockread approach may very well be The
Ultimate Wide Finder.

Timing for tbray_blockread on the 8-core:

real    0m1.546s
user    0m7.337s
sys     0m0.662s

Just starting and stopping the VM takes
> time erl -pa ../../bfile/ebin/ -smp -noshell -run init stop
> real    0m1.229s
> user    0m0.208s
> sys     0m0.020s

On the 8-core this takes:

real    0m1.093s
user    0m0.072s
sys     0m0.012s

It would be interesting to see how it runs on other machines,
> with more cores.

Tim Bray is traveling at the moment, but he told me by email that he hopes
to get back to measuring these on the T5120 next week.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20071023/44a147e2/attachment.html>

More information about the erlang-questions mailing list