<div dir="ltr"><br><br><div class="gmail_quote">On Thu, Jul 24, 2008 at 1:35 PM, Lev Walkin <<a href="mailto:vlm@lionet.info">vlm@lionet.info</a>> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div class="Ih2E3d">hask ellian wrote:<br>
> I made a simple spamfilter in Erlang. It takes 2 files with previous<br>
> spam and good emails and then counts how many times the most frequent<br>
> words from the spammy emails and the good emails occurs and then<br>
> calculates the quote spam/(spam+good) in the file you want to test and<br>
> returns a number between 0 and 1.<br>
> It could easily be improved in numerous ways but the main point for me<br>
> was to learn Erlang. This isn't exactly what Erlang is for but it s way<br>
> to get started.<br>
> I'd be happy to receive comments on the Erlang-ness of the code and<br>
> improvements.<br>
> File I/O seems slow, is there a better way? In Haskell it is fairly instant.<br>
><br>
><br>
> -module(antispam).<br>
> -export([take/2,count/2,count_all/1,most_common/2,count_set_in_list/2,<br>
> classify/0,readfile/1]).<br>
><br>
> take(N,List) -> i_take(N,List,0,[]).<br>
> i_take(N,List,Count,Acc) -><br>
> if Count < N andalso List /= [] -><br>
> i_take(N,tl(List),Count+1,Acc++[hd(List)]);<br>
> Count == N -><br>
> Acc;<br>
> true -><br>
> []<br>
> end.<br>
<br>
</div>Consider using lists:sublist() instead of reimplementing<br>
the standard library methods.<br>
<div class="Ih2E3d"><br>
> count(Tok,List) -> i_count(Tok,List,0).<br>
> i_count(Tok,List,Acc) -><br>
> if Tok == hd(List) andalso List /= [] -><br>
> i_count(Tok,tl(List),Acc+1);<br>
> Tok /= hd(List) andalso List /= [] -><br>
> i_count(Tok,tl(List),Acc);<br>
> true -><br>
> Acc<br>
> end.<br>
<br>
</div>Consider using<br>
<br>
length([T == Tok || T <- List])</blockquote><div><br>I guess you mean<br>count(Tok, List) -> length([T || T<-List, T=:=Tok]).<br><br>or may be better (less memory consuming)<br><br>count(Tok, List) -> lists:foldl(fun(Tok, Acc)->Acc+1; (_,Acc)->Acc end, 0, List).<br>
<br>but tail recursive version is not so bad at all (guess fastest)<br><br>count(Tok, List) -> i_count(Tok, List, 0).<br>i_count(_, [], Count) -> Count;<br>i_count(Tok, [Tok|Rest], Count) -> i_count(Tok, Rest, Count+1);<br>
i_count(Tok, [_|Rest], Count) -> i_count(Tok, Rest, Count).<br><br>and non tail recursive version can work for no so much big Lists reasonably well<br><br>count(_, [])->0;<br>count(Tok, [Tok|Rest])->1+count(Tok, Rest);<br>
count(Tok, [_|Rest)->count(Tok, Rest).<br><br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><br>
<br>
which is much more concise and manageable.<br>
<div class="Ih2E3d"><br>
> count_all(List) -><br>
> Unique = lists:usort(List),<br>
> [{U, count(U, List)} || U <- Unique].<br>
<br>
</div>This one has O(N^2) complexity. Consider using ets if the data<br>
set is larger than, say, 1000 elements.<br>
<div class="Ih2E3d"><br>
> count_set_in_list(Set,List) -><br>
> S = [{S, count(S, List)} || S <- Set],<br>
> lists:sum(lists:map(fun({H,T}) -> T end, S)).<br>
<br>
count_set_in_list(Set, List) -><br>
</div> lists:sum([count(S, List) || S <- Set]).</blockquote><div><br>It is O(N*M) ets or dict should perform better too.<br><br>count_set_in_list(Set, List) -><br> lists:foldl(fun(X, D)->dict:update_counter(X, 1, D) end, dict:new(), List),<br>
lists:foldl(fun(K, Acc)-><br> try dict:fetch(K, CountAll) of<br> Count -> Acc+Count<br> catch _:_->Acc end<br> end, 0, Set).<br><br>or less memory consuming and probalbly faster<br>
<br>count_set_in_list(Set, List) -><br> S = sets:from_list(Set);<br> lists:foldl(fun(K, Acc) -><br> case sets:is_element(K, S) of<br> true -> Acc+1;<br> false -> Acc<br> end<br>
end, 0, List).<br><br>But for short Set it can be slower than <br><br>count_set_in_list(Set, List) -><br> lists:foldl(fun(K, Acc) -><br>
case lists:member(K, Set) of<br>
true -> Acc+1;<br>
false -> Acc<br>
end<br>
end, 0, List).<br><br>which can be simplified as again O(N*M) <br><br>count_set_in_list(Set, List) -><br> length([ok || X<-List, Y<-Set, X=:=Y]).<br><br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>
<div class="Ih2E3d"><br>
> most_common(Stringlist,Xmost) -><br>
> No_preps = lists:filter(fun(X) -> length(X) > 4 end, Stringlist),<br>
> Sorted_by_count = lists:keysort(2, count_all(No_preps)),<br>
> TakeX = take(Xmost, lists:reverse(Sorted_by_count)),<br>
> lists:map(fun({H,T}) -> H end, TakeX).<br>
<br>
<br>
<br>
> readfile(FileName) -><br>
> {ok, Binary} = file:read_file(FileName),<br>
> string:tokens(binary_to_list(Binary), " ").<br>
><br>
> classify() -><br>
> GoodWords =<br>
> most_common(readfile("C:/Users/saftarn/Desktop/emails/okemails.txt"), 20),<br>
> BadWords =<br>
> most_common(readfile("C:/Users/saftarn/Desktop/emails/spam.txt"), 20),<br>
> GoodCount = count_set_in_list(GoodWords,<br>
> readfile("C:/Users/saftarn/Desktop/emails/test.txt")),<br>
> BadCount = count_set_in_list(BadWords,<br>
> readfile("C:/Users/saftarn/Desktop/emails/test.txt")),<br>
> T = BadCount + GoodCount,<br>
> if T /= 0 -><br>
> BadCount / T;<br>
> true -><br>
> 0.5<br>
> end.<br>
><br>
><br>
</div>> ------------------------------------------------------------------------<br>
><br>
> _______________________________________________<br>
> erlang-questions mailing list<br>
> <a href="mailto:erlang-questions@erlang.org">erlang-questions@erlang.org</a><br>
> <a href="http://www.erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://www.erlang.org/mailman/listinfo/erlang-questions</a><br>
<br>
_______________________________________________<br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org">erlang-questions@erlang.org</a><br>
<a href="http://www.erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://www.erlang.org/mailman/listinfo/erlang-questions</a><br>
</blockquote></div><br><br clear="all"><br>-- <br>--Hynek (Pichi) Vychodil
</div>