[erlang-questions] Strings - deprecated functions

Wed Nov 22 22:55:32 CET 2017

Hi All,

Hardly a point in responses so far that I could disagree with or argue against. Although I still contend that the name "lexeme" is about as ugly a term as I could imagine.

So a challenge:

Can someone please write a short tutorial that shows on a one-to-one basis how to use Unicode functions to replace the Latin-1 functions then provide a link to it from the string docs. Or, better yet, simply integrate it into the string docs.

Nevertheless, I'm still deeply troubled.

Many thanks to all.

Lloyd

-----Original Message-----
From: "Dan Gudmundsson" <dgud@REDACTED>
Sent: Wednesday, November 22, 2017 4:45pm
To: "Grzegorz Junka" <list1@REDACTED>
Cc: erlang-questions@REDACTED
Subject: Re: [erlang-questions] Strings - deprecated functions

On Wed, Nov 22, 2017 at 8:59 PM Grzegorz Junka <[ list1@REDACTED ]( mailto:list1@REDACTED )> wrote:

Dear Lloyd,
Isn't this more about documentation than the code? What I am reading is that you want to keep the old functions because you don't understand how the new functions work. Shouldn't you rather ask for a more clear documentation? Is there anything in the old functions that is not supported in the new functions?
GrzegorzJ

On 22/11/2017 19:43, [ lloyd@REDACTED ]( mailto:lloyd@REDACTED ) wrote:
Dear Gods of Erlang,

"This module has been reworked in Erlang/OTP 20 to handle [ unicode:chardata() ]( http://erlang.org/doc/man/unicode.html#type-chardata ) and operate on grapheme clusters. The [ old functions ]( http://erlang.org/doc/man/string.html#oldapi ) that only work on Latin-1 lists as input are still available but should not be used. They will be deprecated in Erlang/OTP 21."

The new functions also works on binaries (or even unicode:chardata() which basically are io-lists but with unicode support) 
which is in many cases a better representation of strings.
That means you can append two long strings with ["long string 1..", "long string 2.."] 
and the string will be "flattened" when output to a file or socket.
So you can see the string module as an introduction how you should handle strings in erlang efficiently. :-)
Though, that said, many of the new functions are slower then the functionality the are replacing, 
optimizations for the ASCII/Latin-1 case are being worked on.
I changed the api rather drastically I know that.
The reason was that in the old 'C' inspired API, you searched first and returned an index,
 then split the string on that index.
Since handling unicode, grapheme clusters, binaries and deep lists of "characters" increases the cost of traversing
the input string, I went with a new API which combines the two calls into one. e.g. find() or take() which searches the string
and returns the result directly, to avoid the extra traversal.

I'm sorry. I've brought up this issue before and got lots of push back.

But every time I look up tried and true and long-used string functions to find that they are deprecated and will be dropped in future Erlang releases my blood pressure soars. Both my wife and my doctor tell me that at my age this is a dangerous thing.

I do understand the importance and necessity of Unicode. And applaud the addition of Unicode functions.

But the deprecated string functions have a long history. The English language and Latin-1 characters are widely used around the world. 

Yes, it should be easy for programmers to translate code from one user language to another. But I'm not convinced that the Gods of Erlang have found the optimal solution by dropping all Latin-1 string functions.
We have not said we will drop them, only deprecate them. They will stay with us for a long time.
I want to remove them from the docs, because the manual page becomes a monster, but we will see what happens with that idea.

My particular application is directed toward English speakers. So, until further notice, I have no use for Unicode.

I don't want to sound like nationalist pig, but I think dropping the Latin-1 string functions from future Erlang releases is a BIG mistake.

I look up tokens/2, a function that I use fairly frequently, and I see that it's deprecated. I look up the suggested replacement and I see lexemes/2.

So I ask, what the ... is a lexeme? I look it up in Merriam-Webster and I see that a lexeme is  "a meaningful linguistic unit." 

Meaning what? I just want to turn "this and that" into "This And That."
So I had to google a lot to come up with a "tokens" replacement function name, a lexeme is what you think a token is or at least what I thought it was. 
see [ https://en.wikipedia.org/wiki/Lexical_analysis ]( https://en.wikipedia.org/wiki/Lexical_analysis )

But 'tokens' is replaced by 'lexemes' and work exactly the same for ASCII lists as before with one exception [CR,LF] but more on that
below.

I read further in the Erlang docs and I see "grapheme cluster."  WHAT THE ... IS GRAPHEME CLUSTER?
This is best description that I found and what I wrote in the manual page:
  A grapheme cluster is a user-perceived character, which can be represented by several codepoints.
I don't know if I can explain it better than that, that is the term used in the unicode standard, more information can be found there.
Below that line I have the self explaining example:

"å"  [229] or [97, 778]"e̊"  [101, 778]
So in Swedish we have the user-perceived character å which is a 'a' with a dot above, 
that can be represented in unicode with codepoint 229 (å) or with the two codepoints  97 (a) 778 (dot above).
So with that we can make "new" combined characters, as I tried with the 'e' and a dot above, which for
me google chrome does not render correctly, the dot should be placed directly above the 'e'.
This representation is important to avoid a "character" explosion for non LATIN-1 character sets such as in Asian and Arabic languages.
You can change between the representations of 'å' with unicode:characters_to_nf[c|d]_list.
It is important that you normalize your data you get from the outside world to one representation before operating on it.
But what they (the Unicode people) did was also define [CR,LF] as one grapheme which makes it impossible to use/extend the old 
functions in compatible way.
So to split a multi-line string into lines, where you previously did:
Lines = string:tokens(" a \n b \r\n c", "\r\n"),    %% Split line on CR or LF
You must now rewrite that to:
Lines = string:lexemes(""a \n b \r\n c", ["\r\n",$n]),  %% Split line on CR,LF or LF
Blame the standard and not me :-)

I look up "grapheme" in Merriam-Webster. Oh it is now all so clear: "a unit of a writing system."

Ah yes, grapheme is defined in the docs. But I have to read and re-read the definition to understand what the God's of Erlang mean by a "graphene cluster." And I'm still not sure I get it.

It sounds like someone took a linguistics class and is trying to show off.
As you can see in this email my linguistic knowledge are way worse than yours, so maybe you can help and improve the manual.
But it is tough to describe unicode handling in an easy way. 
When a character is not a character anymore it becomes fuzzy fast..

Everything here is from the top of my head and not tested, it's too late for that here and I have a Zelda boss to beat, my kids
are way ahead of me.
BR
/Dan

But now I've spent 30 minutes--- time that I don't have to waste trying to figure out how do a simple manipulation of "this and that." Recurse the next time I want to look up a string function in the Erlang docs.

SOLUTION

Keep the Latin-1 string functions. Put them in a separate library if necessary. Or put the new Unicode functions in a separate library. But don't arbitrarily drop them.

Some folks have suggested that I maintain my own library of the deprecated Latin1 functions. But why should I have to do that? How does that help other folks with the same issue?

Bottom line: please please please do not drop the existing Latin-1 string functions.

Please don't.

Best wishes,

LRP

_______________________________________________erlang-questions mailing list[ erlang-questions@REDACTED ]( mailto:erlang-questions@REDACTED )[ http://erlang.org/mailman/listinfo/erlang-questions ]( http://erlang.org/mailman/listinfo/erlang-questions )_______________________________________________
 erlang-questions mailing list
[ erlang-questions@REDACTED ]( mailto:erlang-questions@REDACTED )
[ http://erlang.org/mailman/listinfo/erlang-questions ]( http://erlang.org/mailman/listinfo/erlang-questions )
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20171122/3238ea8d/attachment.htm>