<br><br><div class="gmail_quote">On Wed, Oct 19, 2011 at 6:14 AM, Joe Armstrong <span dir="ltr"><<a href="mailto:erlang@gmail.com">erlang@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
cookbook # 1 - draft 1<br>
<br>
<aside><br>
We're going to write a cookbook.<br>
<br>
This will be free (in an electronic version, PDF, epub)<br>
And you will be able to buy a paper version (POD)<br>
<br>
The development model is<br>
<br>
- a few authors<br>
- many reviewers (you are the reviewers)<br>
the reviewers report errors/suggest changes<br>
the authors make the changes<br></blockquote><div><br></div><div>That sounds neat.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<br>
The POD version we hope will generate some income<br>
this will be split according to the contributions. Authors<br>
will be paid as will reviewers whose suggestions are incorporated.<br>
<br>
Payment (if we make a profit) will be in direct relation to the size<br>
of the contribution<br>
<br>
Expensive things like professional proof reading, will be<br>
sponsorship, or crowd sourced, or otherwise financed.<br>
<br>
To start the ball rolling I have some text below.<br>
<br>
Please comment on this text. If your comments are accepted one day you<br>
might get paid :-)<br>
<br>
Note: 1) By commenting you are implicitly agreeing that if your comments<br>
are accepted into the final text then you will be subject to the<br>
licensing conditions of that text. The text will always be free and<br>
open source.<br>
<br>
</aside><br>
<br>
Cookbook Question:<br>
<br>
I have often seen the words "UTF-8 string" used in sentences like<br>
"Java has UTF-8 strings". What does this mean when applied to Erlang?<br>
<br>
----------------------------------------------------------------------<br>
<br>
Answer:<br>
<br>
In Erlang strings are syntactic sugar for "lists of integers"<br>
<br>
Imagine the string "10(Euro)" - (Euro) is the glyph representing the<br>
Euro currency symbol.<br>
<br>
The term "UF8-string" representing "10(euro)" in Erlang could<br>
mean one of two things:<br>
<br>
Either a) [49,48,8364] (ie its a list of three unicode integers)<br>
Or b) [49,48,226,130,172] (ie its the UTF-8 encoding of the<br>
unicode characters)<br>
<br>
The so words "UTF-8" string might mean a) or might mean b)<br></blockquote><div><br></div><div>No, it won't mean b). See this (using ~ts instead of ~s to specify we want unicode handling):</div><div><br></div>
<div>4> io:format("~ts~n",[<<49,48,226,130,172>>]).</div><div>10€</div><div>ok</div><div>5> io:format("~ts~n",[[49,48,226,130,172]]).</div><div>10€</div><div>ok</div><div>6> io:format("~ts~n",[[49,48,8364]]).</div>
<div>10€</div><div>ok</div><div><br></div><div>They are not the same thing. See <a href="http://ferd.ca/will-the-real-unicode-wrangler-please-stand-up.html">http://ferd.ca/will-the-real-unicode-wrangler-please-stand-up.html</a> for a bit of stuff I've written on it, although my terminology in that blog post is far from stellar and exact according to unicode standards.</div>
<div><br></div><div>The gist of it is that binaries and strings do not share the same ways to represent unicode strings. Unicode strings will have full-length codepoints, and binaries will have the byte-based representation you show.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<br>
Erlang folks have always said "unicode/UTF-8 is easy in Erlang<br>
since strings are just lists of integers" - by this we mean that<br>
Erlang programs should always manipulate strings given the type a)<br>
interpretation. *all* library functions assume type a) encoding.<br></blockquote><div><br></div><div>Not always. list_to_binary and binary_to_list won't work well with that. You need to use unicode:characters_to_[binary|list]/1-3 and make sure the original string has been encoded correctly.</div>
<div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<br>
The type b) interpretation only has meaning when you write data to a<br>
file etc. and should be as invisible to the user as possible (but when<br>
things go wrong and you get the wrong character printed you need to<br>
understand the difference)<br></blockquote><div><br></div><div>This can work when printing to a file because you can print raw bytes using a list, but it's rather hard and I would prefer people to push the distinction above (binary for bytes, lists for codepoints and numbers larger than bytes)</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<br>
Question 1) How can we get a unicode characters into a list item?<br>
or what does a string literal look like?<br>
<br>
> X = "10\x{20ac}"<br>
[49,48,8364]<br></blockquote><div><br></div><div>This is one way to do it, yes. You can also copy/paste the character directly in the shell if I recall.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<br>
This is not described in my book since the change came after the<br>
book was published (is it in the other Erlang books yet?)<br></blockquote><div><br></div><div>Not that I know of. </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<br>
Question 2) How can we convert between representations a) and b) above?<br>
<br>
Easy - though one has to dig in the documentation a bit.<br>
<br>
> B = unicode:characters_to_binary(X, unicode, utf8).<br>
<<49,48,226,130,172>><br>
> unicode:characters_to_list(B).<br>
[49,48,8364]<br></blockquote><div><br></div><div>Yep. </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<br>
Question 3) Can I write "10(Euro)" in an editor which supports<br>
unicode/UTF-8 and does the erlang tool chain support this?<br>
<br>
Will "erlc foo.erl" automatically detect that foo.erl is unicode<br>
encoded and do the right thing when scanning and tokenising strings?<br>
<br>
Answer: I don't know?<br></blockquote><div><br></div><div>Nope. Erlang assumes all files to be in latin-1. Anything that looks like unicode support is luck at best. There is no good Unicode support from static files, although the shell does support it.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<br>
Question 4) Can string literals be improved on?<br>
<br>
I hope so -- In Html I can say (I hope) €<br>
<br>
I'd like to say:<br>
<br>
X = "10€" in Erlang<br>
<br>
People who know far more about this than I do can tell me if this<br>
is OK<br></blockquote><div><br></div><div>This might be funny because people might already use that escaping for HTML content on site. If you start inlining that one and converting to unicode symbols for them, there's no telling how backwards compatible this will be. In any case, I'd well prefer the compiler to support utf-8,utf-16 or utf-32 encoded files than support for '&<code>;' or '&#ijk;' encoding. I'm pretty sure the non-latin-1 users of this mailing list would agree there, too.</div>
<div><br></div></div>