<p dir="ltr">Sorry. I took your statement out of context. </p>
<div class="gmail_quote">On Aug 2, 2012 12:40 AM, "Richard O'Keefe" <<a href="mailto:ok@cs.otago.ac.nz">ok@cs.otago.ac.nz</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
On 2/08/2012, at 3:18 PM, Eric Moritz wrote:<br>
<br>
><br>
> > There is no byte sequence valid in UTF-8 that is not also<br>
> > valid in Latin-1.<br>
><br>
> This is incorrect.<br>
<br>
Let's be pedantic here.<br>
There is no sequence of bytes B such that<br>
(1) B conforms to the rules of UTF-8 and<br>
(2) B can also be decoded as Latin 1<br>
<br>
This is 100% correct.<br>
><br>
> Latin-1 code points are a subset of Unicode codepoints.<br>
<br>
True and totally irrelevant. The statement in question has<br>
nothing to say about codepoints.<br>
<br>
> Codepoints are not bytes.<br>
<br>
Also true and totally irrelevant. The statement in question<br>
has nothing to say about codepoints.<br>
<br>
> Codepoints are indexes in character tables. latin-1 is a table of a possible 256 characters where as Unicode is at this point a table of more that 100,000 characters. There are actually codepoints in the range of 127-159 which are unused and if used are technically invalid Latin-1 and Unicode.<br>
<br>
I suppose it depends on what you mean by "Latin 1".<br>
If you look at the code tables in<br>
<a href="http://www.ecma-international.org/publications/files/ECMA-ST/Ecma-094.pdf" target="_blank">http://www.ecma-international.org/publications/files/ECMA-ST/Ecma-094.pdf</a><br>
you're right: 127 is not there.<br>
<br>
But then neither are TAB, CR, or LF there.<br>
<br>
If you want to talk about "Latin 1" in any sense that includes<br>
those control characters, you have to admit the others.<br>
The framework is specified by ECMA 43, which requires ESC and<br>
DEL. So byte 127 is not invalid.<br>
If you want TAB, CR, LF, and so on, then you get them from<br>
ECMA 48, the C0 set. Bytes with values 128 to 159 *also* come from<br>
ECMA 48.<br>
<br>
So when I talk about "Latin 1" I mean all the printing characters<br>
*and* all the ECMA C0 and C1 control characters.<br>
<br>
It's not just me. Look for example at<br>
<a href="http://www.madore.org/~david/computers/unicode/cstab.html#Latin-1" target="_blank">http://www.madore.org/~david/computers/unicode/cstab.html#Latin-1</a><br>
which shows the control character names in red.<br>
More importantly, look at the mapping tables produced by the<br>
Unicode consortium, specifically 8859-1.TXT.<br>
0x7F 0x007F # DELETE<br>
0x80 0x0080 # <control><br>
...<br>
0x9F 0x009F # <control><br>
0xA0 0x00A0 # NO-BREAK SPACE<br>
<br>
The Unicode consortium think that 0x7F to 0x9F are Latin-1<br>
control characters -- they use #UNDEFINED to mark characters<br>
that are not defined at all in the source character set --<br>
and for what it's worth, U+007F to U+009F are listed in the<br>
Unicode character data base as *defined* characters with<br>
class Cc, and they formerly even named the functions they<br>
perform.<br>
><br>
> When it comes to the binary representation of these codepoints.<br>
<br>
I specifically wrote about BYTE SEQUENCES. Nothing else is<br>
relevant. I did not write about codepoints.<br>
<br>
> Latin-1 is encoded as literal bytes because all codepoints are less than 256.<br>
<br>
You can encode Latin 1 in all sorts of ways.<br>
Bytes work because it's a member of the<br>
ECMA "8-Bit Coded Character Set" family.'<br>
<br>
> Unicode codepoints on the other hand can be larger than 255 so in order to represent them as bytes they need to be encoded.<br>
<br>
That's not relevant. It doesn't matter *what* UTF-8 encodes here,<br>
the only point is that since a UTF-8 sequence is a byte sequence,<br>
and since every byte sequence is a valid Latin 1 encoding, there<br>
is no byte sequence that is a valid UTF-8 sequence but not a valid<br>
Latin 1 sequence.<br>
<br>
There are of course many ways to encode Unicode as sequences of bytes.<br>
We could, to be ridiculous, represent each Unicode codepoint as a<br>
sequence of 21 bytes each with value 0x30 or 0x31. More realistically,<br>
SCSU and BOCU have advantages. The thing is, there is no byte<br>
sequence that cannot be interpreted as representing a sequence of<br>
Latin 1 characters (including control characters), so there is no way<br>
of being certain what you have.<br>
<br>
Of course an XML document must start with zero or more white space<br>
characters followed by a left angle bracket. A higher level protocol<br>
like that _may_ impose constraints that let you figure out what you<br>
have. Similarly an Erlang module must start with a zero or more<br>
white space characters or % comments followed by a hyphen-minus character.<br>
That is enough to allow XML-style discrimination between big- and<br>
little-endian 4-byte and 2-byte representations, some flavour of<br>
EBCDIC, and some extension of ASCII, but not to discriminate between<br>
Latin 1 and UTF-8.<br>
<br>
I've deleted the rest of the message as also beside the point.<br>
<br>
</blockquote></div>