<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Hi Richard,<div><br></div><div>Thanks for the analysis.</div><div>KJB was simply choosen to simplify the problem. Maybe you can suggest a better one as long as it'll exceed +4MB size (and up to 100MB).</div><div><br></div><div>My main concern now is to find out a speedy way for words detection (i.e [{offset, length} | ...]).</div><div><br></div><div><div>Regards,</div><div>Zabrane</div></div><div><br></div><div><div><blockquote type="cite"><div>There are some oddities about that file. For example, it appears to be<br>a document with CR+LF line endings compressed as binary, so that -a does<br>not remove the CRs. There are stray "#" characters for no apparent<br>reason. And above all, verses are written one per line with leading<br>verse *numbers*, except for Philemon, where the first two verses are in<br>a single line. The chapter and verse numbers are part of the *structure*<br>of the document, not part of the *contents*. There _are_ as it happens<br>some numbers in the text (but not in the actual Bible, in the preamble)<br>which one might want to search for.<br><br><blockquote type="cite">$ erl<br></blockquote><blockquote type="cite"><blockquote type="cite">{ok, Bin} = file:read_file("AV1611Bible.txt").<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">{ok, MP} = re:compile("<a href="smb://w+">\\w+</a>").<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">{match, L} = re:run(Bin, MP, [global]).<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">length(L).<br></blockquote></blockquote><blockquote type="cite">839979<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">When timing this version on my machine, I got:<br></blockquote><blockquote type="cite"><blockquote type="cite">timer:tc( fun() -> re:run(Bin, MP, [global]) end ).<br></blockquote></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">2002007 us., which is OK.<br></blockquote><blockquote type="cite">But can we do better? And how fast we can go?<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">The word separators for this problem are: $\s, $\t, $\n, and $\r.<br></blockquote><br>Not counting \n and \r, I get this list:<br> 806901 ' '<br> 72022 ','<br> 26593 '.'<br> 21533 ']'<br> 21533 '['<br> 12748 ':'<br> 10312 ';'<br> 3360 '?'<br> 2950 '#'<br> 2105 '''<br> 896 '-'<br> 315 '!'<br> 306 ')'<br> 306 '('<br> 141 '"'<br> 2 '>'<br> 2 '<'<br> 1 '&'<br><br>One <> pair comes from "b<ishop>" and another from "<of>".<br>For your purposes, you probably want to discard those characters completely.<br>The use of the square brackets in the preamble is distinctly odd.<br>I suspect that whoever prepared this text intended [] to mark the use of italics.<br>You might well want to treat "--" and "-" differently: someone looking for<br>"Bethlehem" might be disappointed to miss "Beth-lehem".<br><br><blockquote type="cite">You can use anything you'd like to accelerate the solution (binary matching, os:cmd(), open_port, NIF, LinkedIn driver).<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">The only one constraint is to get back a list of [{offset, size} | ...] as a result.<br></blockquote><br>If you have an n-core machine, you could simply chop the binary into n pieces,<br>find words in each of them, adjust the results (a bit like how you do parallel<br>prefix on a coarse-grained machine), and paste the results together. In fact,<br>keeping them separate might do even better, depending on what you want to DO<br>with the results.<br><br>m% ctime mawk -f idct.awk AV1611Bible.txt <br>804918 occurrences of 13579 distinct words.<br>0.570 user + 0.070 system = 0.640 total in 1.074 real seconds.<br>This ignores alphabetic case.<br><br>For many text processing purposes, a list of words is better than a<br>list of byte slices. One possible representation of a document is<br> - a tuple of binaries (or strings, or atoms, or what you please)<br> saying what the distinct words _are_<br> (and if that tuple is not in alphabetic order, you might want to<br> have a hash table inverting it)<br> - a list (or tuple, or tree, or what you please) of small integers<br> indexing into the dictionary, saying which words are where.<br>It is certainly going to be a lot quicker to search for a phrase in<br>such a representation.<br><blockquote type="cite"><br></blockquote><blockquote type="cite">Waiting your hacks ... thanks!!! <br></blockquote><br></div></blockquote></div><br><div apple-content-edited="true">
<span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><br></div><div><br></div></div></span></div></span></span><br class="Apple-interchange-newline">
</div>
<br></div></body></html>