[erlang-questions] Strings - deprecated functions

[Dan Gudmundsson]

On Wed, Nov 22, 2017 at 8:59 PM Grzegorz Junka <list1@REDACTED> wrote:

> Dear Lloyd,
>
> Isn't this more about documentation than the code? What I am reading is
> that you want to keep the old functions because you don't understand how
> the new functions work. Shouldn't you rather ask for a more clear
> documentation? Is there anything in the old functions that is not supported
> in the new functions?
>
> GrzegorzJ
>
> On 22/11/2017 19:43, lloyd@REDACTED wrote:
>
> Dear Gods of Erlang,
>
>
>
> "This module has been reworked in Erlang/OTP 20 to handle
> unicode:chardata() <http://erlang.org/doc/man/unicode.html#type-chardata>
> and operate on grapheme clusters. The old functions
> <http://erlang.org/doc/man/string.html#oldapi> that only work on Latin-1
> lists as input are still available but should not be used. They will be
> deprecated in Erlang/OTP 21."
>
>
>
>
The new functions also works on binaries (or even unicode:chardata() which
basically are io-lists but with unicode support)
which is in many cases a better representation of strings.

That means you can append two long strings with ["long string 1..", "long
string 2.."]
and the string will be "flattened" when output to a file or socket.

So you can see the string module as an introduction how you should handle
strings in erlang efficiently. :-)
Though, that said, many of the new functions are slower then the
functionality the are replacing,
optimizations for the ASCII/Latin-1 case are being worked on.

I changed the api rather drastically I know that.
The reason was that in the old 'C' inspired API, you searched first and
returned an index,
 then split the string on that index.

Since handling unicode, grapheme clusters, binaries and deep lists of
"characters" increases the cost of traversing
the input string, I went with a new API which combines the two calls into
one. e.g. find() or take() which searches the string
and returns the result directly, to avoid the extra traversal.

I'm sorry. I've brought up this issue before and got lots of push back.
>
>
>
> But every time I look up tried and true and long-used string functions to
> find that they are deprecated and will be dropped in future Erlang releases
> my blood pressure soars. Both my wife and my doctor tell me that at my age
> this is a dangerous thing.
>
>
>
> I do understand the importance and necessity of Unicode. And applaud the
> addition of Unicode functions.
>
>
>
> But the deprecated string functions have a long history. The English
> language and Latin-1 characters are widely used around the world.
>
>
>
> Yes, it should be easy for programmers to translate code from one user
> language to another. But I'm not convinced that the Gods of Erlang have
> found the optimal solution by dropping all Latin-1 string functions.
>
>
We have not said we will drop them, only deprecate them. They will stay
with us for a long time.
I want to remove them from the docs, because the manual page becomes a
monster, but we will see what happens with that idea.


>
>
> My particular application is directed toward English speakers. So, until
> further notice, I have no use for Unicode.
>
>
>
> I don't want to sound like nationalist pig, but I think dropping the
> Latin-1 string functions from future Erlang releases is a BIG mistake.
>
>
>
> I look up tokens/2, a function that I use fairly frequently, and I see
> that it's deprecated. I look up the suggested replacement and I see
> lexemes/2.
>
>
>
> So I ask, what the ... is a lexeme? I look it up in Merriam-Webster and I
> see that a lexeme is  "a meaningful linguistic unit."
>
>
>
> Meaning what? I just want to turn "this and that" into "This And That."
>
>
So I had to google a lot to come up with a "tokens" replacement function
name, a lexeme is what you think a token is or at least what I thought it
was.
see https://en.wikipedia.org/wiki/Lexical_analysis

But 'tokens' is replaced by 'lexemes' and work exactly the same for ASCII
lists as before with one exception [CR,LF] but more on that
below.

>
>
> I read further in the Erlang docs and I see "grapheme cluster."  WHAT THE
> ... IS GRAPHEME CLUSTER?
>
> This is best description that I found and what I wrote in the manual page:
  A *grapheme cluster* is a user-perceived character, which can be
represented by several codepoints.

I don't know if I can explain it better than that, that is the term used in
the unicode standard, more information can be found there.
Below that line I have the self explaining example:

"å"  [229] or [97, 778]"e̊"  [101, 778]


So in Swedish we have the user-perceived character å which is a 'a' with a
dot above,
that can be represented in unicode with codepoint 229 (å) or with the two
codepoints  97 (a) 778 (dot above).

So with that we can make "new" combined characters, as I tried with the 'e'
and a dot above, which for
me google chrome does not render correctly, the dot should be placed
directly above the 'e'.

This representation is important to avoid a "character" explosion for non
LATIN-1 character sets such as in Asian and Arabic languages.

You can change between the representations of 'å' with
unicode:characters_to_nf[c|d]_list.
It is important that you normalize your data you get from the outside world
to one representation before operating on it.

But what they (the Unicode people) did was also define [CR,LF] as one
grapheme which makes it impossible to use/extend the old
functions in compatible way.

So to split a multi-line string into lines, where you previously did:
Lines = string:tokens(" a \n b \r\n c", "\r\n"),    %% Split line on CR or
LF
You must now rewrite that to:
Lines = string:lexemes(""a \n b \r\n c", ["\r\n",$n]),  %% Split line on
CR,LF or LF

Blame the standard and not me :-)


>
> I look up "grapheme" in Merriam-Webster. Oh it is now all so clear: "a
> unit of a writing system."
>
>
>
> Ah yes, grapheme is defined in the docs. But I have to read and re-read
> the definition to understand what the God's of Erlang mean by a "graphene
> cluster." And I'm still not sure I get it.
>
>
>
> It sounds like someone took a linguistics class and is trying to show off.
>
>
As you can see in this email my linguistic knowledge are way worse than
yours, so maybe you can help and improve the manual.
But it is tough to describe unicode handling in an easy way.
When a character is not a character anymore it becomes fuzzy fast..

Everything here is from the top of my head and not tested, it's too late
for that here and I have a Zelda boss to beat, my kids
are way ahead of me.

BR
/Dan


>
>
> But now I've spent 30 minutes--- time that I don't have to waste trying to
> figure out how do a simple manipulation of "this and that." Recurse the
> next time I want to look up a string function in the Erlang docs.
>
>
>
> SOLUTION
>
>
>
> Keep the Latin-1 string functions. Put them in a separate library if
> necessary. Or put the new Unicode functions in a separate library. But
> don't arbitrarily drop them.
>
>
>
> Some folks have suggested that I maintain my own library of the deprecated
> Latin1 functions. But why should I have to do that? How does that help
> other folks with the same issue?
>
>
>
> Bottom line: please please please do not drop the existing Latin-1 string
> functions.
>
>
>
> Please don't.
>
>
>
> Best wishes,
>
>
>
> LRP
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> _______________________________________________
> erlang-questions mailing listerlang-questions@REDACTED://erlang.org/mailman/listinfo/erlang-questions
>
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20171122/03a76f5c/attachment.htm>