<div dir="ltr"><br><br><div class="gmail_quote"><div dir="ltr">On Wed, Nov 22, 2017 at 8:59 PM Grzegorz Junka <<a href="mailto:list1@gjunka.com">list1@gjunka.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<p>Dear Lloyd,</p>
<p>Isn't this more about documentation than the code? What I am
reading is that you want to keep the old functions because you
don't understand how the new functions work. Shouldn't you rather
ask for a more clear documentation? Is there anything in the old
functions that is not supported in the new functions?</p>
<p>GrzegorzJ<br>
</p></div><div bgcolor="#FFFFFF" text="#000000">
<br>
<div class="m_-6983078336347844642moz-cite-prefix">On 22/11/2017 19:43,
<a class="m_-6983078336347844642moz-txt-link-abbreviated" href="mailto:lloyd@writersglen.com" target="_blank">lloyd@writersglen.com</a> wrote:<br>
</div>
<blockquote type="cite"><font face="arial" size="2">
<p style="margin:0;padding:0;font-family:arial;font-size:10pt">Dear Gods of Erlang,</p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt">"This module has been
reworked in Erlang/OTP 20 to handle <span class="m_-6983078336347844642bold_code m_-6983078336347844642bc-15"><a href="http://erlang.org/doc/man/unicode.html#type-chardata" target="_blank"><span class="m_-6983078336347844642code">unicode:chardata()</span></a></span>
and operate on grapheme clusters. The <span class="m_-6983078336347844642bold_code m_-6983078336347844642bc-17"><a href="http://erlang.org/doc/man/string.html#oldapi" target="_blank"><span class="m_-6983078336347844642code">old functions</span></a></span>
that only work on Latin-1 lists as input are still available
but should not be used. They will be deprecated in Erlang/OTP
21."</p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p></font></blockquote></div></blockquote><div><br></div><div>The new functions also works on binaries (or even unicode:chardata() which basically are io-lists but with unicode support) </div><div>which is in many cases a better representation of strings.</div><div><br></div><div>That means you can append two long strings with ["long string 1..", "long string 2.."] </div><div>and the string will be "flattened" when output to a file or socket.</div><div><br></div><div>So you can see the string module as an introduction how you should handle strings in erlang efficiently. :-)</div><div>Though, that said, many of the new functions are slower then the functionality the are replacing, </div><div>optimizations for the ASCII/Latin-1 case are being worked on.</div><div><br></div><div>I changed the api rather drastically I know that.</div><div>The reason was that in the old 'C' inspired API, you searched first and returned an index,</div><div> then split the string on that index.</div><div><br></div><div>Since handling unicode, grapheme clusters, binaries and deep lists of "characters" increases the cost of traversing</div><div>the input string, I went with a new API which combines the two calls into one. e.g. find() or take() which searches the string</div><div>and returns the result directly, to avoid the extra traversal.</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000"><blockquote type="cite"><font face="arial" size="2">
<p style="margin:0;padding:0;font-family:arial;font-size:10pt">I'm sorry. I've brought up
this issue before and got lots of push back.</p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt">But every time I look up
tried and true and long-used string functions to find that
they are deprecated and will be dropped in future Erlang
releases my blood pressure soars. Both my wife and my doctor
tell me that at my age this is a dangerous thing.</p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt">I do understand the
importance and necessity of Unicode. And applaud the addition
of Unicode functions.</p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt">But the deprecated string
functions have a long history. The English language and
Latin-1 characters are widely used around the world. </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt">Yes, it should be easy for
programmers to translate code from one user language to
another. But I'm not convinced that the Gods of Erlang have
found the optimal solution by dropping all Latin-1 string
functions.</p></font></blockquote></div></blockquote><div><br></div><div>We have not said we will drop them, only deprecate them. They will stay with us for a long time.</div><div>I want to remove them from the docs, because the manual page becomes a monster, but we will see what happens with that idea.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000"><blockquote type="cite"><font face="arial" size="2">
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt">My particular application is
directed toward English speakers. So, until further notice, I
have no use for Unicode.</p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt">I don't want to sound like
nationalist pig, but I think dropping the Latin-1 string
functions from future Erlang releases is a BIG mistake.</p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt">I look up tokens/2, a
function that I use fairly frequently, and I see that it's
deprecated. I look up the suggested replacement and I see
lexemes/2.</p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt">So I ask, what the ... is a
lexeme? I look it up in Merriam-Webster and I see that a
lexeme is "a meaningful linguistic unit." </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt">Meaning what? I just want to
turn "this and that" into "This And That."</p></font></blockquote></div></blockquote><div><br></div><div>So I had to google a lot to come up with a "tokens" replacement function name, a lexeme is what you think a token is or at least what I thought it was. </div><div>see <a href="https://en.wikipedia.org/wiki/Lexical_analysis">https://en.wikipedia.org/wiki/Lexical_analysis</a></div><div> <br></div><div>But 'tokens' is replaced by 'lexemes' and work exactly the same for ASCII lists as before with one exception [CR,LF] but more on that</div><div>below.</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000"><blockquote type="cite"><font face="arial" size="2"><p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt">I read further in the Erlang
docs and I see "grapheme cluster." WHAT THE ... IS GRAPHEME
CLUSTER?</p></font></blockquote></div></blockquote><div>This is best description that I found and what I wrote in the manual page:</div><div> <span style="color:rgb(26,26,26);font-size:16px;background-color:rgb(254,254,254)"><font face="monospace"> </font>A </span><strong style="color:rgb(26,26,26);font-size:16px;background-color:rgb(254,254,254)">grapheme cluster</strong><span style="color:rgb(26,26,26);font-size:16px;background-color:rgb(254,254,254)"> is a user-perceived character, which can be represented by several codepoints.</span></div><div><br></div><div>I don't know if I can explain it better than that, that is the term used in the unicode standard, more information can be found there.</div><div>Below that line I have the self explaining example:</div><div><br></div><div><pre style="font-family:mono,Courier,monospace;margin-top:0px;margin-bottom:0px;color:rgb(56,58,66);font-size:11.2px"><span class="inbox-inbox-hljs-string" style="color:rgb(80,161,79)">"å"</span> [<span class="inbox-inbox-hljs-number" style="color:rgb(152,104,1)">229</span>] or [<span class="inbox-inbox-hljs-number" style="color:rgb(152,104,1)">97</span>, <span class="inbox-inbox-hljs-number" style="color:rgb(152,104,1)">778</span>]
<span class="inbox-inbox-hljs-string" style="color:rgb(80,161,79)">"e̊"</span> [<span class="inbox-inbox-hljs-number" style="color:rgb(152,104,1)">101</span>, <span class="inbox-inbox-hljs-number" style="color:rgb(152,104,1)">778</span>]</pre></div><div><br></div><div>So in Swedish we have the user-perceived character å which is a 'a' with a dot above, </div><div>that can be represented in unicode with codepoint 229 (å) or with the two codepoints 97 (a) 778 (dot above).</div><div><br></div><div>So with that we can make "new" combined characters, as I tried with the 'e' and a dot above, which for</div><div>me google chrome does not render correctly, the dot should be placed directly above the 'e'.</div><div><br></div><div>This representation is important to avoid a "character" explosion for non LATIN-1 character sets such as in Asian and Arabic languages.</div><div><br></div><div>You can change between the representations of 'å' with unicode:characters_to_nf[c|d]_list.</div><div>It is important that you normalize your data you get from the outside world to one representation before operating on it.</div><div><br></div><div>But what they (the Unicode people) did was also define [CR,LF] as one grapheme which makes it impossible to use/extend the old </div><div>functions in compatible way.</div><div><br></div><div>So to split a multi-line string into lines, where you previously did:</div><div>Lines = string:tokens(" a \n b \r\n c", "\r\n"), %% Split line on CR or LF<br></div><div>You must now rewrite that to:</div><div>Lines = string:lexemes(""a \n b \r\n c", ["\r\n",$n]), %% Split line on CR,LF or LF</div><div><br></div><div>Blame the standard and not me :-)</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000"><blockquote type="cite"><font face="arial" size="2">
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt">I look up "grapheme" in
Merriam-Webster. Oh it is now all so clear: "a unit of a
writing system."</p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt">Ah yes, grapheme is defined
in the docs. But I have to read and re-read the definition to
understand what the God's of Erlang mean by a "graphene
cluster." And I'm still not sure I get it.</p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt">It sounds like someone took
a linguistics class and is trying to show off.</p></font></blockquote></div></blockquote><div><br></div><div>As you can see in this email my linguistic knowledge are way worse than yours, so maybe you can help and improve the manual.</div><div>But it is tough to describe unicode handling in an easy way. </div><div>When a character is not a character anymore it becomes fuzzy fast..</div><div><br></div><div><div>Everything here is from the top of my head and not tested, it's too late for that here and I have a Zelda boss to beat, my kids</div><div>are way ahead of me.</div></div><div><br></div><div>BR<br>/Dan</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000"><blockquote type="cite"><font face="arial" size="2">
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt">But now I've spent 30
minutes--- time that I don't have to waste trying to figure
out how do a simple manipulation of "this and that." Recurse
the next time I want to look up a string function in the
Erlang docs.</p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt">SOLUTION</p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt">Keep the Latin-1 string
functions. Put them in a separate library if necessary. Or put
the new Unicode functions in a separate library. But don't
arbitrarily drop them.</p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt">Some folks have suggested
that I maintain my own library of the deprecated Latin1
functions. But why should I have to do that? How does that
help other folks with the same issue?</p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt">Bottom line: please please
please do not drop the existing Latin-1 string functions.</p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt">Please don't.</p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt">Best wishes,</p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt">LRP</p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
<p style="margin:0;padding:0;font-family:arial;font-size:10pt"> </p>
</font>
<br>
<fieldset class="m_-6983078336347844642mimeAttachmentHeader"></fieldset>
<br>
<pre>_______________________________________________
erlang-questions mailing list
<a class="m_-6983078336347844642moz-txt-link-abbreviated" href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a>
<a class="m_-6983078336347844642moz-txt-link-freetext" href="http://erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a>
</pre>
</blockquote>
<br>
</div>
_______________________________________________<br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a><br>
<a href="http://erlang.org/mailman/listinfo/erlang-questions" rel="noreferrer" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
</blockquote></div></div>