<div dir="ltr">I have pushed some initial work that creates a separate chunk called AtU8:<div><br></div><div><a href="https://github.com/erlang/otp/compare/master...josevalim:jv-utf8-atom">https://github.com/erlang/otp/compare/master...josevalim:jv-utf8-atom</a><br></div><div><br></div><div>First the atoms from the Atom chunk are loaded, followed by atoms from the new AtU8 chunk. I have, however, stumbled against one complication. When we generate the atom table in the compiler, the indexes are built based on first-seen case. So there is a chance we will have an atom table that looks like this:</div><div><br></div><div><font face="monospace, monospace"> #{0 => module_name,</font></div><div><font face="monospace, monospace"> 1 => 'some_utf8_atom_ł',</font></div><div><font face="monospace, monospace"> 2 => latin1_atom}</font></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature"><div dir="ltr"><div>However, because we first load the latin chunk, the loaded atom table would look like this:</div><div><br></div><div><div><font face="monospace, monospace"> 0 => module_name</font></div><div><span style="font-family:monospace,monospace"> 1 => latin1_atom</span><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"> 2 => 'some_utf8_atom_ł'</font></div><div><br></div></div><div>Which won't work. I have thought of a couple solutions to this problem. I would appreciate some feedback on which one is preferred:</div><div><br></div><div>1. Keep on using two separate chunks and change beam_asm to do two passes. One first collecting the latin1 atoms so we know how to generate indexes for the utf8 ones.</div><div><br></div><div>2. Keep on using two separate chunks and use negative indexes to encode utf8 atoms. This means the compiler will build a table that looks like this:</div><div><br></div><div><div><font face="monospace, monospace"> #{-1 => </font><span style="font-family:monospace,monospace">'some_utf8_atom_ł',</span></div><div><span style="font-family:monospace,monospace"> 0 => module_name</span><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"> 1 => </font><span style="font-family:monospace,monospace">latin1_atom}</span></div></div><div><br></div><div>Which we will load to the same table as before:</div><div><br></div><div><div><font face="monospace, monospace"> 0 => module_name</font></div><div><span style="font-family:monospace,monospace"> 1 => latin1_atom</span><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"> 2 => 'some_utf8_atom_ł'</font></div></div><div><br></div><div>And we will translate negative indexes to the proper position by calculating "num_atoms + index" when loading the bytecode.</div><div><br></div><div>3. Introduce a new chunk called AtoE (Atoms with Encoding) to *replace* the existing Atom chunk. It will be quite similar to the current Atom chunk except we will also include the encoding alongside each atom size and contents. The Atom chunk will only be loaded if there is no AtoE chunk. If we choose this option, we can also choose to always emit the new chunk or emit the AtoE chunk only if there are Unicode atoms.</div><div><br></div><div>4. Similar to option 3) except that we introduce a new Atom chunk that will keep all atoms but always in Unicode. Since the runtime translates them to latin1 later anyway, this is an option if we don't want to store the encoding in the table.</div><div><br></div><div>Thank you!</div><div><br></div><div>PS: If you'd prefer this conversation to be moved off the list, please let me know.</div><div><br></div><div><span style="font-size:13px"><div><span style="font-family:arial,sans-serif;font-size:13px;border-collapse:collapse"><b>José Valim</b></span></div><div><span style="font-family:arial,sans-serif;font-size:13px;border-collapse:collapse"><div><span style="font-family:verdana,sans-serif;font-size:x-small"><a href="http://www.plataformatec.com.br/" style="color:rgb(42,93,176)" target="_blank">www.plataformatec.com.br</a></span></div><div><span style="font-family:verdana,sans-serif;font-size:x-small">Skype: jv.ptec</span></div><div><span style="font-family:verdana,sans-serif;font-size:x-small">Founder and Director of R&D</span></div></span></div></span></div></div></div></div>
<br><div class="gmail_quote">On Mon, Feb 1, 2016 at 1:08 PM, José Valim <span dir="ltr"><<a href="mailto:jose.valim@plataformatec.com.br" target="_blank">jose.valim@plataformatec.com.br</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Understood! Thank you.<div><br></div></div><div class="gmail_extra"><span class=""><br clear="all"><div><div><div dir="ltr"><div><div><br></div><div><br></div><div><span style="font-size:13px"><div><span style="font-family:arial,sans-serif;font-size:13px;border-collapse:collapse"><b>José Valim</b></span></div><div><span style="font-family:arial,sans-serif;font-size:13px;border-collapse:collapse"><div><span style="font-family:verdana,sans-serif;font-size:x-small"><a href="http://www.plataformatec.com.br/" style="color:rgb(42,93,176)" target="_blank">www.plataformatec.com.br</a></span></div><div><span style="font-family:verdana,sans-serif;font-size:x-small">Skype: jv.ptec</span></div><div><span style="font-family:verdana,sans-serif;font-size:x-small">Founder and Director of R&D</span></div></span></div></span></div></div></div></div></div>
<br></span><div><div class="h5"><div class="gmail_quote">On Mon, Feb 1, 2016 at 12:59 PM, Björn Gustavsson <span dir="ltr"><<a href="mailto:bjorn@erlang.org" target="_blank">bjorn@erlang.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span>On Mon, Feb 1, 2016 at 9:44 AM, José Valim<br>
<<a href="mailto:jose.valim@plataformatec.com.br" target="_blank">jose.valim@plataformatec.com.br</a>> wrote:<br>
> So I would say list_to_binary is behaving as expected and that it should not<br>
> change as those "limitations" are there today. Same for port_command, as it<br>
> expects iodata. Or am I missing something?<br>
<br>
</span>My point is that we must look for code in<br>
OTP that will break when the change to<br>
the atoms are made.<br>
<br>
As an hypothetical example, say that we<br>
find the following code in some application:<br>
<br>
Str = atom_to_list(Atom),<br>
.<br>
.<br>
.<br>
port_command(Port, Cmd, Str)<br>
<br>
We must look at the context to determine<br>
what we should do. There could be one<br>
of several solutions, for example:<br>
<br>
1. If the atoms that can be passed to this<br>
code have been internally generated we<br>
could know that the resulting list is always<br>
safe to send to the port. In that case we<br>
don't need to update the code.<br>
<br>
2. If the origin of the atom is unknown,<br>
and the driver cannot handle UTF-8,<br>
the solution could be to return an error<br>
to the caller if the atom contains<br>
non-latin1 characters.<br>
<br>
3. If the driver can handle UTF-8 or can<br>
be modified to handle UTF-8, the solution<br>
could be to use atom_to_binary(Atom, utf8)<br>
instead of atom_to_list/1.<br>
<br>
Basically, we must look at every atom_to_list/1<br>
in the OTP code base and determine whether<br>
it is safe or if it must be modified in some way.<br>
<div><div><br>
/Björn<br>
<br>
--<br>
Björn Gustavsson, Erlang/OTP, Ericsson AB<br>
</div></div></blockquote></div><br></div></div></div>
</blockquote></div><br></div></div>