<div dir="ltr"><br><div class="gmail_quote"><div dir="ltr">On Sat, Oct 14, 2017 at 6:23 PM Alexandre Karpov <<a href="mailto:alexakarpov@gmail.com">alexakarpov@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Thanks everyone! I didn't realize until this conversation how much more important strings-as-binaries are, compared to simple "strings". Everything _works_ now, of course, but I don't think my understanding has caught up 100%<div><br></div><div></div></div><div dir="ltr"><div><div>"by default it guesses that lists containing integers larger than 255</div></div></div><div dir="ltr"><div><div>is not a string but a list of integers" <<< this really set some things straight</div><div><br></div><div>But suffer me this follow-up question, Dan. Using +pc unicode indeed gave me a shell that represents lists of integers using the characters found in Unicode mapping; so now, in error messages, I see arguments reported more clearly:</div><div><br></div><div>
<p class="m_-5719709182889794815gmail-m_-3758754219055318119gmail-p1"><span class="m_-5719709182889794815gmail-m_-3758754219055318119gmail-s1"><b>7> re:run("йцу.asd", xmerl_regexp:sh_to_awk("*.*"), [{capture, none}]). <span class="m_-5719709182889794815gmail-m_-3758754219055318119gmail-Apple-converted-space"> </span></b></span></p></div></div></div><div dir="ltr"><div><div>
<p class="m_-5719709182889794815gmail-m_-3758754219055318119gmail-p1"><span class="m_-5719709182889794815gmail-m_-3758754219055318119gmail-s1"><b>** exception error: bad argument</b></span></p>
<p class="m_-5719709182889794815gmail-m_-3758754219055318119gmail-p1"><span class="m_-5719709182889794815gmail-m_-3758754219055318119gmail-s1"><b><span class="m_-5719709182889794815gmail-m_-3758754219055318119gmail-Apple-converted-space"> </span>in function<span class="m_-5719709182889794815gmail-m_-3758754219055318119gmail-Apple-converted-space"> </span>re:run/3</b></span></p>
</div></div></div><div dir="ltr"><div><div><p class="m_-5719709182889794815gmail-m_-3758754219055318119gmail-p1"><span class="m_-5719709182889794815gmail-m_-3758754219055318119gmail-s1"><b><span class="m_-5719709182889794815gmail-m_-3758754219055318119gmail-Apple-converted-space"> </span>called as re:run("йцу.asd","^(.*\\..*)$",[{capture,none}])</b></span></p>
<p class="m_-5719709182889794815gmail-m_-3758754219055318119gmail-p1"><span class="m_-5719709182889794815gmail-m_-3758754219055318119gmail-s1"><b><br></b></span></p><p class="m_-5719709182889794815gmail-m_-3758754219055318119gmail-p1"><span class="m_-5719709182889794815gmail-m_-3758754219055318119gmail-s1">If I use the binary-string representation, it works _even_without_ /utf8, it works just fine:</span></p><p class="m_-5719709182889794815gmail-p1"><span class="m_-5719709182889794815gmail-s1"><b>3> re:run(<<"普通話.asd">>, xmerl_regexp:sh_to_awk("*.*"), [{capture, none}]).</b></span></p><p class="m_-5719709182889794815gmail-m_-3758754219055318119gmail-p1"><b>
</b></p><p class="m_-5719709182889794815gmail-p1"><span class="m_-5719709182889794815gmail-s1"><b>match</b></span></p></div><div class="gmail_extra">Note that the call above was executed in the shell started _without_ the +pc unicode, and the binary does _not_ have the /utf8>> thingy... This means my understanding is still lacking... binaries are honest and good, strings are fake and evil, but +px unicode seems to help a little with fake string... while using binary-strings doesn't _always_ require the /utf8 ... what is this sorcery?!</div><div class="gmail_extra">=)</div></div></div></blockquote><div><br></div><div>First see the difference here:</div><div><br></div><div><div><b>4> <<"普通話.asd">>.</b></div><div><b><<110,26,113,46,97,115,100>></b></div><div><b>i.e. the codepoints are just truncated to below 256</b></div><div><b><br></b></div><div><b>5> <<"普通話.asd"/utf8>>.</b></div><div><b><<230,153,174,233,128,154,232,169,177,46,97,115,100>></b></div><div><b>6> </b></div></div><div>And the codepoints are utf8 encoded. </div><div><br></div><div>Don't give up on lists, they can be useful and fast for some usages.</div><div><br></div><div>And your regexp matches anything with a dot in it, so even if the string is handled as utf8 encoded binary or just plain bytes, it</div><div>still works since in both representation you get a match.</div><div><br></div><div>To understand unicode, play around and try to make it work (on both lists and binaries) with some fancier regexps try match</div><div>something with a unicode sign in it and capture the result so you see what you matched. </div><div>Print your input/result strings with both io:format("~ts: ~w~n",[Str, Str]). So you can see both the actual string and it's representation,</div><div>test with both binaries and lists as representations.</div><div><br></div><div>/Dan</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div class="gmail_extra"><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Oct 14, 2017 at 4:24 AM, Dan Gudmundsson <span dir="ltr"><<a href="mailto:dangud@gmail.com" target="_blank">dangud@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">re:run("йцу.asd", xmerl_regexp:sh_to_awk("*.*"), [{capture, none}, unicode]).<br><div>The binary one matches since it works on bytes and not utf-8 characters?<br></div><div><br></div><div>Also the erlang shell doesn't know if a list of integers is a list of integers or a string,</div><div>since they may be represented by the same list of integers.</div><div><br></div><div>So it tries to guess, by default it guesses that lists containing integers larger than 255</div><div>is not a string but a list of integers. You can change that with:</div><div><br></div><div>(w)erl +pc unicode</div><div><br></div><div>1> "йцу.asd".</div><div>"йцу.asd"</div><span class="m_-5719709182889794815gmail-m_-3758754219055318119gmail-m_1157110655033658819gmail-HOEnZb"><font color="#888888"><div><br></div><div>/Dan</div><div><br></div></font></span></div><br><div class="gmail_quote"><div><div class="m_-5719709182889794815gmail-m_-3758754219055318119gmail-m_1157110655033658819gmail-h5"><div dir="ltr">On Sat, Oct 14, 2017 at 10:12 AM Attila Rajmund Nohl <<a href="mailto:attila.r.nohl@gmail.com" target="_blank">attila.r.nohl@gmail.com</a>> wrote:<br></div></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="m_-5719709182889794815gmail-m_-3758754219055318119gmail-m_1157110655033658819gmail-h5">2017-10-14 4:21 GMT+02:00 Alexandre Karpov <<a href="mailto:alexakarpov@gmail.com" target="_blank">alexakarpov@gmail.com</a>>:<br>
> TL;DR: how do I run erl which understands Unicode?<br>
><br>
> Or, in more detail:<br>
><br>
> (Disclaimer: this official documentation got me really humbled:<br>
> <a href="http://www1.erlang.org/doc/apps/stdlib/unicode_usage.html" rel="noreferrer" target="_blank">http://www1.erlang.org/doc/apps/stdlib/unicode_usage.html</a><br>
> , and just a little bit scared =) )<br>
><br>
> Judging by my S/O question, which got 3 upvotes and no answers, I'm not the<br>
> only one wondering:<br>
> <a href="https://stackoverflow.com/questions/46735539/erlang-regexp-matching-on-chinese-characters" rel="noreferrer" target="_blank">https://stackoverflow.com/questions/46735539/erlang-regexp-matching-on-chinese-characters</a><br>
><br>
> Here's the gist of the problem:<br>
><br>
> 57> "абв".<br>
><br>
> [1072,1073,1074]<br>
><br>
> The codes are correct Unicode for the [Cyrillic] characters - which means my<br>
> Terminal didn't fail to understand my keyboard's input =) but Erlang shell<br>
> didn't recognize Terminal's input as printable characters. And it is my<br>
> understanding that this is exactly why this call fails:<br>
><br>
> 25> re:run("йцу.asd", xmerl_regexp:sh_to_awk("*.*"), [{capture, none}]). **<br>
> exception error: bad argument in function re:run/3 called as<br>
> re:run([1081,1094,1091,46,97,115,100], "^(.*\\..*)$", [{capture,none}])<br>
<br>
Try<br>
<br>
re:run(<<"йцу.asd"/utf8>>, xmerl_regexp:sh_to_awk("*.*"), [{capture, none}]).<br></div></div><span class="m_-5719709182889794815gmail-m_-3758754219055318119gmail-m_1157110655033658819gmail-">
_______________________________________________<br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a><br>
<a href="http://erlang.org/mailman/listinfo/erlang-questions" rel="noreferrer" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
</span></blockquote></div>
</blockquote></div><br></div></div></div></blockquote></div></div>