<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div style="-webkit-text-size-adjust: auto;">Hi Pichi,</div><div style="-webkit-text-size-adjust: auto;"><br></div><div style="-webkit-text-size-adjust: auto;">Since I haven't learned yet how to design and conduct performance tests, results like these are both interesting and comforting.</div><div style="-webkit-text-size-adjust: auto;"><br></div><div style="-webkit-text-size-adjust: auto;">The long stop words list in <a href="http://www.ranks.nl/stopwords">http://www.ranks.nl/stopwords</a> has something less than 700 words. So from these results it looks like either method would do the job in most applications, unless you are filtering stop words out of a huge archive of long documents.</div><div style="-webkit-text-size-adjust: auto;"><br></div><div style="-webkit-text-size-adjust: auto;">Many thanks, Pichi.</div><div style="-webkit-text-size-adjust: auto;"><br></div><div style="-webkit-text-size-adjust: auto;">Best wishes,</div><div style="-webkit-text-size-adjust: auto;"><br></div><div style="-webkit-text-size-adjust: auto;">LRP</div><div><br><span style="-webkit-text-size-adjust: auto;">Sent from my iPad</span></div><div style="-webkit-text-size-adjust: auto;"><br>On Mar 30, 2016, at 2:12 PM, Hynek Vychodil <<a href="mailto:vychodil.hynek@gmail.com">vychodil.hynek@gmail.com</a>> wrote:<br><br></div><blockquote type="cite" style="-webkit-text-size-adjust: auto;"><div><div dir="ltr">Every time I read a claim about how fast it will be I have urge test it. I had an idea that constant map in a module could be faster than function clause co I test it.<div><br></div><div>I was wrong and RAO is right as usual. Function using function clause seems to be three times faster than using map.</div><div><br></div><div><div>x clause</div><div>+ map</div><div>+--------------------------------------------------------------------------+</div><div>|xxxxx +++++ +|</div><div>|xxxx ++++ |</div><div>|xxxx +++ |</div><div>|xxxx ++ |</div><div>|xxx ++ |</div><div>|xxx ++ |</div><div>|xx ++ |</div><div>|xx ++ |</div><div>|xx ++ |</div><div>|xx + |</div><div>|xx + |</div><div>|xx + |</div><div>|xx + |</div><div>|xx + |</div><div>| x + |</div><div>| x + |</div><div>| x + |</div><div>| x + |</div><div>| x + |</div><div>| x + |</div><div>| x + |</div><div>| x + |</div><div>| x + |</div><div>| x + |</div><div>| x + |</div><div>| + |</div><div>| + |</div><div>| + |</div><div>| + |</div><div>| + |</div><div>| + |</div><div>| + |</div><div>| + |</div><div>| + |</div><div>||A| |</div><div>| |_MA_| |</div><div>+--------------------------------------------------------------------------+</div><div>Dataset: x N=50 CI=95.0000</div><div>Statistic Value [ Bias] (Bootstrapped LB‥UB)</div><div>Min: 3490.00</div><div>1st Qu. 3551.00</div><div>Median: 3591.00</div><div>3rd Qu. 3679.00</div><div>Max: 3945.00</div><div>Average: 3630.16 [ 0.137534] ( 3602.82 ‥ 3664.56)</div><div>Std. Dev: 113.400 [ -1.81311] ( 90.8425 ‥ 141.539)</div><div><br></div><div>Outliers: 0/4 = 4 (μ=3630.30, σ=111.587)</div><div> Outlier variance: 0.151802 (moderate)</div><div><br></div><div>------</div><div><br></div><div>Dataset: + N=50 CI=95.0000</div><div>Statistic Value [ Bias] (Bootstrapped LB‥UB)</div><div>Min: 1.09500e+4</div><div>1st Qu. 1.10160e+4</div><div>Median: 1.10400e+4</div><div>3rd Qu. 1.11270e+4</div><div>Max: 1.28270e+4</div><div>Average: 1.11055e+4 [ 0.297998] ( 1.10611e+4 ‥ 1.12491e+4)</div><div>Std. Dev: 264.914 [ -31.0673] ( 84.7956 ‥ 582.629)</div><div><br></div><div>Outliers: 0/2 = 2 (μ=1.11058e+4, σ=233.847)</div><div> Outlier variance: 9.45082e-2 (slight)</div><div><br></div><div>Difference at 95.0% confidence</div><div> 7475.36 ± 80.8533</div><div> 205.924% ± 2.22726%</div><div> (Student's t, pooled s = 203.763)</div><div>------</div><div><br></div><div>It's about 31 million stopwords_clause:is_stopword/1 per second and 10 million stopwords_map:is_stopword/1 per second.</div><div><br></div><div>You can find code in gist <a href="https://gist.github.com/pichi/2d10c93242d5057913d026a607f07dd4">https://gist.github.com/pichi/2d10c93242d5057913d026a607f07dd4</a></div><div><br></div><div>Pichi</div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Mar 30, 2016 at 4:05 AM, Lloyd R. Prentice <span dir="ltr"><<a href="mailto:lloyd@writersglen.com" target="_blank">lloyd@writersglen.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">Wow! What a cool idea.<br>
<br>
Thanks, Richard.<br>
<br>
Best wishes,<br>
<br>
LRP<br>
<br>
Sent from my iPad<br>
<br>
> On Mar 29, 2016, at 8:47 PM, "Richard A. O'Keefe" <<a href="mailto:ok@cs.otago.ac.nz">ok@cs.otago.ac.nz</a>> wrote:<br>
<span class="">><br>
><br>
>> On 30/03/16 5:59 am, <a href="mailto:lloyd@writersglen.com">lloyd@writersglen.com</a> wrote:<br>
>> So, I have a printed list of stop words:<br>
>><br>
>> <a href="http://www.ranks.nl/stopwords" rel="noreferrer" target="_blank">http://www.ranks.nl/stopwords</a><br>
>><br>
>> I'd like to turn this list into an Erlang function that I can query---<br>
>><br>
>> stopwords() -><br>
>> ["word1", "word2" ... "wordN"].<br>
>><br>
>> is_stopword(Word) -><br>
>> List = stopwords(),<br>
>> lists_member(Word, List).<br>
</span>> Even if there is some arcane reason why you want the collection of words<br>
> as a list, I strongly suggest generating<br>
><br>
> is_stopword("a") -> true;<br>
> is_stopword("about") -> true;<br>
> ...<br>
> is_stopword("yourselves") -> true;<br>
> is_stopword(_) -> false.<br>
><br>
> Open the list of stopwords in vi.<br>
> :1,$s/^.*$/is_stopword("&") -> true;/<br>
> :$a<br>
> is_stopword(_) -> false.<br>
> <ESC><br>
><br>
> The Erlang compiler will turn this into a trie, roughly speaking.<br>
> This will be *dizzyingly* faster than the code you outlined.<br>
<div class=""><div class="h5">><br>
><br>
><br>
><br>
>><br>
>> All my efforts so far have evolved into ugly kludges. Seems to me there must be an elegant method that I'm overlooking.<br>
>><br>
>> Some kind soul point the way?<br>
>><br>
>> Many thanks,<br>
>><br>
>> LRP<br>
>><br>
>> *********************************************<br>
>> My books:<br>
>><br>
>> THE GOSPEL OF ASHES<br>
>> <a href="http://thegospelofashes.com" rel="noreferrer" target="_blank">http://thegospelofashes.com</a><br>
>><br>
>> Strength is not enough. Do they have the courage<br>
>> and the cunning? Can they survive long enough to<br>
>> save the lives of millions?<br>
>><br>
>> FREEIN' PANCHO<br>
>> <a href="http://freeinpancho.com" rel="noreferrer" target="_blank">http://freeinpancho.com</a><br>
>><br>
>> A community of misfits help a troubled boy find his way<br>
>><br>
>> AYA TAKEO<br>
>> <a href="http://ayatakeo.com" rel="noreferrer" target="_blank">http://ayatakeo.com</a><br>
>><br>
>> Star-crossed love, war and power in an alternative<br>
>> universe<br>
>><br>
>> Available through Amazon or by request from your<br>
>> favorite bookstore<br>
>><br>
>><br>
>> **********************************************<br>
>><br>
>> _______________________________________________<br>
>> erlang-questions mailing list<br>
>> <a href="mailto:erlang-questions@erlang.org">erlang-questions@erlang.org</a><br>
>> <a href="http://erlang.org/mailman/listinfo/erlang-questions" rel="noreferrer" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
><br>
_______________________________________________<br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org">erlang-questions@erlang.org</a><br>
<a href="http://erlang.org/mailman/listinfo/erlang-questions" rel="noreferrer" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
</div></div></blockquote></div><br></div></div></div>
</div></blockquote></body></html>