<div dir="ltr">In this case, the words comes partly from terms you would find in linguistics, partly words which have specific meaning in the unicode standard.<div><br></div><div>The problem with Latin-1 and ISO8859-1 and ISO8859-15 are that they work somewhat well for Western Latin languages, but it falls short on almost everything else. If your only concern is truly English text, then there should be no worry at all, since that uses ASCII and the predominant Unicode enconding, UTF-8 was chosen such that there is a 1-1 overlap between the first 128 characters and ASCII.</div><div><br></div><div>However, Unicode imposes some difficulties. The most notable one is that you have several ways of writing symbols such as the danish Ø and Å: Either as a specific character, or as a combination: and A and a small ring on top for instance.</div><div><br></div><div>In languages the written symbols are graphemes, and collections of symbols forming tokens or words are lexemes. However, because one grapheme can be represented as one or several characters, the notion of a grapheme cluster arises: several code-points which form a single grapheme. It is of utmost importance for certain Asian writing systems in which a single grapheme is composed out of several smaller ones.</div><div><br></div><div>For ASCII, however, string:lexemes/2 would work exactly like string:tokens/2. Yet it will handle far more cases.</div><div><br></div><div>Unicode presents its own set of complexities. There are several ways of writing a unicode string which is "the same" string in that it renders equally to the human eye. Hence, there are some routines for handling normalization, canonization and collation which by no means are easy to handle.</div><div><br></div><div>And finally, it would probably be good to define those terms in the documentation. I don't think they are well-known to most people. </div></div><br><div class="gmail_quote"><div dir="ltr">On Wed, Nov 22, 2017 at 8:59 PM Grzegorz Junka <<a href="mailto:list1@gjunka.com">list1@gjunka.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<p>Dear Lloyd,</p>
<p>Isn't this more about documentation than the code? What I am
reading is that you want to keep the old functions because you
don't understand how the new functions work. Shouldn't you rather
ask for a more clear documentation? Is there anything in the old
functions that is not supported in the new functions?</p>
<p>GrzegorzJ<br>
</p></div><div bgcolor="#FFFFFF" text="#000000">
<br>
<div class="m_-1143854075824632532moz-cite-prefix">On 22/11/2017 19:43,
<a class="m_-1143854075824632532moz-txt-link-abbreviated" href="mailto:lloyd@writersglen.com" target="_blank">lloyd@writersglen.com</a> wrote:<br>
</div>
<blockquote type="cite"><font face="arial" size="2">
<p style="margin:0;padding:0;font-family:arial;font-size:10pt">Dear Gods of Erlang,</p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt">"This module has been
reworked in Erlang/OTP 20 to handle <span class="m_-1143854075824632532bold_code m_-1143854075824632532bc-15"><a href="http://erlang.org/doc/man/unicode.html#type-chardata" target="_blank"><span class="m_-1143854075824632532code">unicode:chardata()</span></a></span>
and operate on grapheme clusters. The <span class="m_-1143854075824632532bold_code m_-1143854075824632532bc-17"><a href="http://erlang.org/doc/man/string.html#oldapi" target="_blank"><span class="m_-1143854075824632532code">old functions</span></a></span>
that only work on Latin-1 lists as input are still available
but should not be used. They will be deprecated in Erlang/OTP
21."</p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt">I'm sorry. I've brought up
this issue before and got lots of push back.</p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt">But every time I look up
tried and true and long-used string functions to find that
they are deprecated and will be dropped in future Erlang
releases my blood pressure soars. Both my wife and my doctor
tell me that at my age this is a dangerous thing.</p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt">I do understand the
importance and necessity of Unicode. And applaud the addition
of Unicode functions.</p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt">But the deprecated string
functions have a long history. The English language and
Latin-1 characters are widely used around the world. </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt">Yes, it should be easy for
programmers to translate code from one user language to
another. But I'm not convinced that the Gods of Erlang have
found the optimal solution by dropping all Latin-1 string
functions.</p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt">My particular application is
directed toward English speakers. So, until further notice, I
have no use for Unicode.</p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt">I don't want to sound like
nationalist pig, but I think dropping the Latin-1 string
functions from future Erlang releases is a BIG mistake.</p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt">I look up tokens/2, a
function that I use fairly frequently, and I see that it's
deprecated. I look up the suggested replacement and I see
lexemes/2.</p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt">So I ask, what the ... is a
lexeme? I look it up in Merriam-Webster and I see that a
lexeme is "a meaningful linguistic unit." </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt">Meaning what? I just want to
turn "this and that" into "This And That." </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt">I read further in the Erlang
docs and I see "grapheme cluster." WHAT THE ... IS GRAPHEME
CLUSTER?</p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt">I look up "grapheme" in
Merriam-Webster. Oh it is now all so clear: "a unit of a
writing system."</p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt">Ah yes, grapheme is defined
in the docs. But I have to read and re-read the definition to
understand what the God's of Erlang mean by a "graphene
cluster." And I'm still not sure I get it.</p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt">It sounds like someone took
a linguistics class and is trying to show off.</p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt">But now I've spent 30
minutes--- time that I don't have to waste trying to figure
out how do a simple manipulation of "this and that." Recurse
the next time I want to look up a string function in the
Erlang docs.</p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt">SOLUTION</p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt">Keep the Latin-1 string
functions. Put them in a separate library if necessary. Or put
the new Unicode functions in a separate library. But don't
arbitrarily drop them.</p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt">Some folks have suggested
that I maintain my own library of the deprecated Latin1
functions. But why should I have to do that? How does that
help other folks with the same issue?</p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt">Bottom line: please please
please do not drop the existing Latin-1 string functions.</p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt">Please don't.</p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt">Best wishes,</p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt">LRP</p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
</font>
<br>
<fieldset class="m_-1143854075824632532mimeAttachmentHeader"></fieldset>
<br>
<pre>_______________________________________________
erlang-questions mailing list
<a class="m_-1143854075824632532moz-txt-link-abbreviated" href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a>
<a class="m_-1143854075824632532moz-txt-link-freetext" href="http://erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a>
</pre>
</blockquote>
<br>
</div>
_______________________________________________<br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a><br>
<a href="http://erlang.org/mailman/listinfo/erlang-questions" rel="noreferrer" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
</blockquote></div>