[erlang-questions] widefinder update

[Anders Nygren]

To summarize my progress on the widefinder problem
A few days ago I started with Steve Vinoski's tbray16.erl
As a baseline on my 1.66 GHz dual core Centrino
laptop, Linux,
tbray16
real    0m7.067s
user    0m12.377s
sys     0m0.584s

I removed the dict used for the shift table,
and changed the min_heap_size.
That gave
real    0m2.713s
user    0m4.168s
sys     0m0.412s

(see tbray_tuple.erl and wfbm4_tuple.erl)
Steve reported that it ran in ~1.9 s on his 8 core server.

Then I removed the dicts that were used for collecting the
matches and used ets instead, and got some improvement
on my dual core laptop.
real    0m2.220s
user    0m3.252s
sys     0m0.344s

(see tbray_ets.erl and wfbm4_ets.erl)

Interestingly Steve reported that it actually performed
worse on his 8 core server.

These versions all read the whole file into memory at the start.
On my laptop that takes ~400ms (when the file is already cached
in the OS).

So I changed it to read the file in chucks and spawn the worker
after each chunk is read.

tbray_blockread with 4 processes
real    0m1.992s
user    0m3.176s
sys     0m0.420s

(see tbray_blockread.erl and wfbm4_ets.erl)

Running it in the erlang shell it takes ~1.8s.

Just starting and stopping the VM takes
time erl -pa ../../bfile/ebin/ -smp -noshell -run init stop

real    0m1.229s
user    0m0.208s
sys     0m0.020s

It would be interesting to see how it runs on other machines,
with more cores.

/Anders
-------------- next part --------------
A non-text attachment was scrubbed...
Name: tbray16_tuple.erl
Type: text/x-erlang
Size: 2490 bytes
Desc: not available
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20071023/71115e7c/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: wfbm4_tuple.erl
Type: text/x-erlang
Size: 3497 bytes
Desc: not available
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20071023/71115e7c/attachment-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: tbray_ets.erl
Type: text/x-erlang
Size: 2622 bytes
Desc: not available
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20071023/71115e7c/attachment-0002.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: wfbm4_ets.erl
Type: text/x-erlang
Size: 3848 bytes
Desc: not available
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20071023/71115e7c/attachment-0003.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: tbray_blockread.erl
Type: text/x-erlang
Size: 3308 bytes
Desc: not available
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20071023/71115e7c/attachment-0004.bin>