<div dir="ltr">Also as far as I remember icu works with utf16, so any call to icu requires encoding binary, which is usually usually utf8, to utf16 and then encoding it back to utf8 binary.</div><div class="gmail_extra"><br><div class="gmail_quote">2017-01-14 19:06 GMT+03:00 Benoit Chesneau <span dir="ltr"><<a href="mailto:bchesneau@gmail.com" target="_blank">bchesneau@gmail.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><br><div class="gmail_quote"><span class=""><div dir="ltr">On Sat, Jan 14, 2017 at 4:53 PM Oliver Korpilla <<a href="mailto:Oliver.Korpilla@gmx.de" target="_blank">Oliver.Korpilla@gmx.de</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Could the Unicode support in elixir serve as a starting point?<br class="m_4762514505458294205gmail_msg">
<br class="m_4762514505458294205gmail_msg">
<a href="https://hexdocs.pm/elixir/1.3.3/String.html#content" rel="noreferrer" class="m_4762514505458294205gmail_msg" target="_blank">https://hexdocs.pm/elixir/1.3.<wbr>3/String.html#content</a><br class="m_4762514505458294205gmail_msg">
<br class="m_4762514505458294205gmail_msg">
String.upcase/1 and String.downcase/1 seem to be Unicode-aware. And a lot of effort seems have gone in scenarios like this:<br class="m_4762514505458294205gmail_msg">
<br class="m_4762514505458294205gmail_msg">
"For example, the codepoint “é” is two bytes:<br class="m_4762514505458294205gmail_msg">
<br class="m_4762514505458294205gmail_msg">
iex> byte_size("é")<br class="m_4762514505458294205gmail_msg">
2"<br class="m_4762514505458294205gmail_msg">
<br class="m_4762514505458294205gmail_msg">
Given that both Erlang and elixir are implemented on top of BEAM, the wheel might not need reinventing? I know engineers and programmers love inventing stuff, and this discussion seems to point in that direction, but...<br class="m_4762514505458294205gmail_msg">
<br class="m_4762514505458294205gmail_msg">
Cheers,<br class="m_4762514505458294205gmail_msg">
Oliver<br class="m_4762514505458294205gmail_msg">
<br class="m_4762514505458294205gmail_msg"></blockquote><div><br></div></span><div>If I remember correctly the unicode support of Elixir is written in elixir and data come from the unicode/icu projects. data resources (codepoints and so on ) are compiled as beam. (I do the dame in my idna lib).<br><br></div><div>The work may be simpler in using/wrting a nif over the well supported ICU lib thoug. I'm curious about the reasonning that conducted to the current implementation in elixir.<span class="HOEnZb"><font color="#888888"><br><br></font></span></div><span class="HOEnZb"><font color="#888888"><div>- benoit<br></div></font></span><div><div class="h5"><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br class="m_4762514505458294205gmail_msg">
<br class="m_4762514505458294205gmail_msg">
Gesendet: Freitag, 13. Januar 2017 um 23:34 Uhr<br class="m_4762514505458294205gmail_msg">
Von: "Michał Muskała" <<a href="mailto:michal@muskala.eu" class="m_4762514505458294205gmail_msg" target="_blank">michal@muskala.eu</a>><br class="m_4762514505458294205gmail_msg">
An: "Richard A. O'Keefe" <<a href="mailto:ok@cs.otago.ac.nz" class="m_4762514505458294205gmail_msg" target="_blank">ok@cs.otago.ac.nz</a>>, "Steve Davis" <<a href="mailto:steven.charles.davis@gmail.com" class="m_4762514505458294205gmail_msg" target="_blank">steven.charles.davis@gmail.<wbr>com</a>>, <a href="mailto:g@rre.tt" class="m_4762514505458294205gmail_msg" target="_blank">g@rre.tt</a>, "Jesper Louis Andersen" <<a href="mailto:jesper.louis.andersen@gmail.com" class="m_4762514505458294205gmail_msg" target="_blank">jesper.louis.andersen@gmail.<wbr>com</a>><br class="m_4762514505458294205gmail_msg">
Cc: "Erlang Questions" <<a href="mailto:erlang-questions@erlang.org" class="m_4762514505458294205gmail_msg" target="_blank">erlang-questions@erlang.org</a>><br class="m_4762514505458294205gmail_msg">
Betreff: Re: [erlang-questions] Erlang basic doubts about String, message passing and context switching overhead<br class="m_4762514505458294205gmail_msg">
<br class="m_4762514505458294205gmail_msg">
I fully agree there are no languages that deal with strings perfectly. That said there are those that are better at it and those that aren't so good. A language, where I need to look for a library to upcase or downcase my own name, fits into the second group in my book.<br class="m_4762514505458294205gmail_msg">
<br class="m_4762514505458294205gmail_msg">
Michał.<br class="m_4762514505458294205gmail_msg">
On 13 Jan 2017, 13:20 +0100, Jesper Louis Andersen <<a href="mailto:jesper.louis.andersen@gmail.com" class="m_4762514505458294205gmail_msg" target="_blank">jesper.louis.andersen@gmail.<wbr>com</a>>, wrote:<br class="m_4762514505458294205gmail_msg">
<br class="m_4762514505458294205gmail_msg">
Richard is indeed right, depending on what your definition of "String" is.<br class="m_4762514505458294205gmail_msg">
If a "String" is "An array of characters from some alphabet", then you need to take into account Strings are Unicode codepoints in practice. This is also the most precise definition from a technical point of view.<br class="m_4762514505458294205gmail_msg">
When I wrote my post, I was--probably incorrectly--assuming the older notion of a "String" where the representation is either ASCII or something like ISO-8859-15. In this case, a string coincides with a stream of bytes.<br class="m_4762514505458294205gmail_msg">
Data needs parsing. A lot of data comes in as some kind of stringy representation: UTF-8, byte array (binary), and so on.<br class="m_4762514505458294205gmail_msg">
And of course, that isn't the whole story, since there are examples of input which are not string-like in their forms.<br class="m_4762514505458294205gmail_msg">
<br class="m_4762514505458294205gmail_msg">
<br class="m_4762514505458294205gmail_msg">
On Fri, Jan 13, 2017 at 2:34 AM Richard A. O'Keefe <<a href="mailto:ok@cs.otago.ac.nz" class="m_4762514505458294205gmail_msg" target="_blank">ok@cs.otago.ac.nz</a>[mailto:<a href="mailto:ok@cs.otago.ac.nz" class="m_4762514505458294205gmail_msg" target="_blank">ok@<wbr>cs.otago.ac.nz</a>]> wrote:<br class="m_4762514505458294205gmail_msg">
<br class="m_4762514505458294205gmail_msg">
On 13/01/17 8:56 AM, Jesper Louis Andersen wrote:<br class="m_4762514505458294205gmail_msg">
> Strings are really just streams of bytes.<br class="m_4762514505458294205gmail_msg">
<br class="m_4762514505458294205gmail_msg">
That was true a long time ago. Maybe.<br class="m_4762514505458294205gmail_msg">
But it isn't anywhere near accurate as a description<br class="m_4762514505458294205gmail_msg">
of Unicode:<br class="m_4762514505458294205gmail_msg">
- Unicode is made of 21-bit code points, not bytes.<br class="m_4762514505458294205gmail_msg">
- Most possible code points are not defined.<br class="m_4762514505458294205gmail_msg">
- Some of those that are defined are defined as<br class="m_4762514505458294205gmail_msg">
"it is illegal to use this".<br class="m_4762514505458294205gmail_msg">
- Unicode sequences have *structure*; it is simply<br class="m_4762514505458294205gmail_msg">
not the case that every sequence of allowable<br class="m_4762514505458294205gmail_msg">
Unicode code points is a legal Unicode string.<br class="m_4762514505458294205gmail_msg">
- As a special case of that, if s is a non-empty<br class="m_4762514505458294205gmail_msg">
valid Unicode string, it is not true that every<br class="m_4762514505458294205gmail_msg">
substring of s is a valid Unicode string.<br class="m_4762514505458294205gmail_msg">
<br class="m_4762514505458294205gmail_msg">
In case you were thinking of UTF-8, not all byte<br class="m_4762514505458294205gmail_msg">
sequences are valid UTF-8.<br class="m_4762514505458294205gmail_msg">
<br class="m_4762514505458294205gmail_msg">
Byte streams are as important as you say, but it's<br class="m_4762514505458294205gmail_msg">
really hard to see the software for a radar or a<br class="m_4762514505458294205gmail_msg">
radio telescope as processing strings...<br class="m_4762514505458294205gmail_msg">
_____________________________<wbr>__________________<br class="m_4762514505458294205gmail_msg">
erlang-questions mailing list<br class="m_4762514505458294205gmail_msg">
<a href="mailto:erlang-questions@erlang.org" class="m_4762514505458294205gmail_msg" target="_blank">erlang-questions@erlang.org</a><br class="m_4762514505458294205gmail_msg">
<a href="http://erlang.org/mailman/listinfo/erlang-questions_______________________________________________" rel="noreferrer" class="m_4762514505458294205gmail_msg" target="_blank">http://erlang.org/mailman/<wbr>listinfo/erlang-questions_____<wbr>______________________________<wbr>____________</a> erlang-questions mailing list <a href="mailto:erlang-questions@erlang.org" class="m_4762514505458294205gmail_msg" target="_blank">erlang-questions@erlang.org</a> <a href="http://erlang.org/mailman/listinfo/erlang-questions%5Bhttp://erlang.org/mailman/listinfo/erlang-questions%5D" rel="noreferrer" class="m_4762514505458294205gmail_msg" target="_blank">http://erlang.org/mailman/<wbr>listinfo/erlang-questions[<wbr>http://erlang.org/mailman/<wbr>listinfo/erlang-questions]</a><br class="m_4762514505458294205gmail_msg">
______________________________<wbr>_________________<br class="m_4762514505458294205gmail_msg">
erlang-questions mailing list<br class="m_4762514505458294205gmail_msg">
<a href="mailto:erlang-questions@erlang.org" class="m_4762514505458294205gmail_msg" target="_blank">erlang-questions@erlang.org</a><br class="m_4762514505458294205gmail_msg">
<a href="http://erlang.org/mailman/listinfo/erlang-questions" rel="noreferrer" class="m_4762514505458294205gmail_msg" target="_blank">http://erlang.org/mailman/<wbr>listinfo/erlang-questions</a><br class="m_4762514505458294205gmail_msg">
</blockquote></div></div></div></div>
<br>______________________________<wbr>_________________<br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org">erlang-questions@erlang.org</a><br>
<a href="http://erlang.org/mailman/listinfo/erlang-questions" rel="noreferrer" target="_blank">http://erlang.org/mailman/<wbr>listinfo/erlang-questions</a><br>
<br></blockquote></div><br></div>