<div dir="ltr"><div class="gmail_default" style="font-family:monospace,monospace">The question means something like this.</div><div class="gmail_default" style="font-family:monospace,monospace">You are to construct and then use a table</div><div class="gmail_default" style="font-family:monospace,monospace"> source : checksum -> set of filename</div><div class="gmail_default" style="font-family:monospace,monospace">by doing</div><div class="gmail_default" style="font-family:monospace,monospace"> for each filename given</div><div class="gmail_default" style="font-family:monospace,monospace"> open the file</div><div class="gmail_default" style="font-family:monospace,monospace"> for each 40-character window of the contents of the file</div><div class="gmail_default" style="font-family:monospace,monospace"> compute the checksum of the window</div><div class="gmail_default" style="font-family:monospace,monospace"> add filename to source{checkum}</div><div class="gmail_default" style="font-family:monospace,monospace"> close the file</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace"> s := {}</div><div class="gmail_default" style="font-family:monospace,monospace"> open the test data</div><div class="gmail_default" style="font-family:monospace,monospace"> for each 40-character window of the contents of the testdata</div><div class="gmail_default" style="font-family:monospace,monospace"> compute the checksum of the window</div><div class="gmail_default" style="font-family:monospace,monospace"> let s' be source{checksum} with default {}</div><div class="gmail_default" style="font-family:monospace,monospace"> s := s U s'</div><div class="gmail_default" style="font-family:monospace,monospace"> close the test data</div><div class="gmail_default" style="font-family:monospace,monospace"> report the set of possible source file names s</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">Now the test is to accomplish, in Erlang, the same</div><div class="gmail_default" style="font-family:monospace,monospace">effect as that would, without necessarily doing it</div><div class="gmail_default" style="font-family:monospace,monospace">that way.</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">This is actually a fairly simplistic version of an</div><div class="gmail_default" style="font-family:monospace,monospace">Information Retrieval task called Passage Retrieval.</div><div class="gmail_default" style="font-family:monospace,monospace">With a little more work, you can not only report</div><div class="gmail_default" style="font-family:monospace,monospace">that the test data may contain portions of various</div><div class="gmail_default" style="font-family:monospace,monospace">files, but *which* portions. With a bit more work</div><div class="gmail_default" style="font-family:monospace,monospace">than that, you can rank them, distinguishing files</div><div class="gmail_default" style="font-family:monospace,monospace">that contribute many passages from ones that contribute</div><div class="gmail_default" style="font-family:monospace,monospace">few. The neat thing here is that, done with care,</div><div class="gmail_default" style="font-family:monospace,monospace">you can handle very large document collections in</div><div class="gmail_default" style="font-family:monospace,monospace">Erlang without as much work as you might have to do in<br></div><div class="gmail_default" style="font-family:monospace,monospace">languages lacking a native-data-structures persistent store.</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">When you look at this functionally, you see that this is</div><div class="gmail_default" style="font-family:monospace,monospace">basically a Map (over files) Reduce (the set unions) and</div><div class="gmail_default" style="font-family:monospace,monospace">that the index building could be done in a distributed</div><div class="gmail_default" style="font-family:monospace,monospace">way, again with relatively little effort. (The set<br></div><div class="gmail_default" style="font-family:monospace,monospace">unions I'm talking about are the "add ... to ..." ones.</div><div class="gmail_default" style="font-family:monospace,monospace">This is idempotent, so indexing the same document on two</div><div class="gmail_default" style="font-family:monospace,monospace">nodes is wasteful but not dangerous.)</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div></div><br><div class="gmail_quote"><div dir="ltr">On Wed, 31 Oct 2018 at 18:55, カカキ <<a href="mailto:heturing@gmail.com">heturing@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr">Hello, everyone.<div><br></div><div>I'm a little confused with an Exercise, and I cannot understand the question.</div><div><br></div><div>Here is the question.</div><div><br></div><div>Write a program to detect plagiarisms in text. To do this, use a two-pass algorithm. In pass 1, break the text into 40-character blocks and compute a checksum for each 40-character block. Store the checksum and filename in an ETS table. In pass 2, compute the checksum of each 40-character block in the data and compare with the checksums in the ETS table.</div><div><br></div><div>Hint: You will need to compute a "rolling checksum" to do this. For example, if C1 = B1 + B2 + ... + B40 and C2 = B2 + B3+ ... + B41, then C2 can be quickly computed by obversing that C2 = C1 + B41 - B1.</div><div><br></div><div>What I've done is finding two files, say file1.txt and file2.txt. I want to check whether file2 plagiarizes file1. So I get first 40-characters of file1 and calculate the checksum of it. And then I move the block one character to the right and calculate the checksum. At last, I will get a list of checksum of file1, and I store them into an ETS table.</div><div><br></div><div>On pass two, I do the same thing with file2 and get a list of checksum [C1, C2, ..., Cn]. For each element, I check whether it exists in ETS table or not. If exists, it means that there is a block with the same content in file 1. So it is plagiarism. Finally, I count the number of these blocks and output it.</div><div><br></div><div>However, it seems that I haven't use the filename. So I am wondering that there is something wrong with my opinion.</div><div><br></div><div>Thanks.</div><div><br></div></div></div></div>
_______________________________________________<br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a><br>
<a href="http://erlang.org/mailman/listinfo/erlang-questions" rel="noreferrer" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
</blockquote></div>