<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Then I suggest you use an other json module that support unicode OR<div class="">scan the xml structure and reformat the text values into UTF while</div><div class="">you are converting the XML into internal JSON structure?</div><div class="">Must be super simple!</div><div class=""><br class=""></div><div class="">/Tony</div><div class=""><br class=""></div><div class=""><div class=""><div><blockquote type="cite" class=""><div class="">On 18 aug 2015, at 17:29, Alexander Turkin <<a href="mailto:snowwlex@gmail.com" class="">snowwlex@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span style="font-size: 12.8000001907349px;" class="">Why do you want to have utf8 in the values ? unicode is much more<br class=""></span><span style="font-size: 12.8000001907349px;" class="">generic and you select the encoding on the way out!</span></blockquote><div class=""><br class=""></div><div class="">The thing is that I need to convert it to json, and mochijson (and jsx as well) doesn't understand this<span style="font-size: 12.8000001907349px;" class=""><b class=""> iso-10646. </b>And when mochijson gets something not utf-8, it throws `</span><font class=""><span style="font-size:12.8000001907349px" class="">{ucs,{bad_utf8_character_code}}}` error.</span></font></div><div class=""><font class=""><span style="font-size:12.8000001907349px" class=""><br class=""></span></font></div><div class=""><span style="font-size: 12.8000001907349px;" class=""><b class=""><br class=""></b></span></div><div class=""><span style="font-size: 12.8000001907349px;" class=""><b class=""><br class=""></b></span></div></div><div class="gmail_extra"><br class=""><div class="gmail_quote">On 17 August 2015 at 16:26, Tony Rogvall <span dir="ltr" class=""><<a href="mailto:tony@rogvall.se" target="_blank" class="">tony@rogvall.se</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Ok I see.<br class="">
<br class="">
So you expected to find utf8 in the text value instead of the unicode ( 233 is the same in latin1 and unicode btw )<br class="">
But that is not how the xmerl works. It represents the characters in unicode iso-10646.<br class="">
<br class="">
Here is an example to get you a utf8 output. Bin is your binary.<br class="">
<br class="">
Term = binary_to_term(Bin).<br class="">
{Content,_} = xmerl_scan:string(binary_to_list(Term)).<br class="">
UnicodeChars = xmerl:export([Content], xmerl_xml).<br class="">
Utf8Bin = unicode:characters_to_binary(UnicodeChars).<br class="">
<br class="">
I guess you could scan the xml structure (Content) and convert the text values<br class="">
to utf8 strings. But that would complicate the process when you want<br class="">
to format the output.<br class="">
<br class="">
Why do you want to have utf8 in the values ? unicode is much more<br class="">
generic and you select the encoding on the way out!<br class="">
<span class="HOEnZb"><font color="#888888" class=""><br class="">
/Tony<br class="">
</font></span><div class="HOEnZb"><div class="h5"><br class="">
<br class="">
> On 17 aug 2015, at 15:56, Alexander Turkin <<a href="mailto:snowwlex@gmail.com" class="">snowwlex@gmail.com</a>> wrote:<br class="">
><br class="">
> Hey Tony,<br class="">
><br class="">
> Yes, this binary is in utf8 - and it is what it's being fed to xmerl library, which returns it in the other encoding by some reasons.<br class="">
><br class="">
> On 17 August 2015 at 13:55, Tony Rogvall <<a href="mailto:tony@rogvall.se" class="">tony@rogvall.se</a>> wrote:<br class="">
> Sorry for the empty message :-)<br class="">
><br class="">
> But the coding you are looking for is already in your binary.<br class="">
><br class="">
> <<131,109,0,0,0,79,60,63,120,109,108,32,118,101,114,115,<br class="">
> 105,111,110,61,34,49,46,48,34,32,101,110,99,111,100,105,<br class="">
> 110,103,61,34,85,84,70,45,56,34,63,62,60,114,101,115,<br class="">
> 112,111,110,115,101,62,60,118,97,108,117,101,62,82,101,<br class="">
> 110,195,169,60,47,118,97,108,117,101,62,60,47,114,101,<br class="">
> 115,112,111,110,115,101,62>><br class="">
><br class="">
> 195,169 = c3 a9<br class="">
><br class="">
> /Tony<br class="">
><br class="">
><br class="">
>> On 17 aug 2015, at 14:39, Alexander Turkin <<a href="mailto:snowwlex@gmail.com" class="">snowwlex@gmail.com</a>> wrote:<br class="">
>><br class="">
>> Hi Hynek,<br class="">
>><br class="">
>> On 15 August 2015 at 08:30, Hynek Vychodil <<a href="mailto:vychodil.hynek@gmail.com" class="">vychodil.hynek@gmail.com</a>> wrote:<br class="">
>> The same result is in R18 and it it correct result. Letter é has unicode 233 see <a href="http://unicode-table.com/en/#00E9" rel="noreferrer" target="_blank" class="">http://unicode-table.com/en/#00E9</a><br class="">
>><br class="">
>> Yeah, it has U+00E9 (= 233) code point number, but it is coded in 2 bytes in utf8: c3 a9<br class="">
>><br class="">
>> U+00E9 é c3 a9 LATIN SMALL LETTER E WITH ACUTE<br class="">
>><br class="">
>><br class="">
>> (<a href="http://www.utf8-chartable.de/" rel="noreferrer" target="_blank" class="">http://www.utf8-chartable.de/</a>)<br class="">
>><br class="">
>> On Fri, Aug 14, 2015 at 6:23 PM, Éric Pailleau <<a href="mailto:eric.pailleau@wanadoo.fr" class="">eric.pailleau@wanadoo.fr</a>> wrote:<br class="">
>> Hello,<br class="">
>> Please precise what Erlang release you are using. Utf8 came lately in Erlang.<br class="">
>> Regards<br class="">
>><br class="">
>> Le 14 août 2015 13:49, Alexander Turkin <<a href="mailto:snowwlex@gmail.com" class="">snowwlex@gmail.com</a>> a écrit :<br class="">
>> ><br class="">
>> > Dear list,<br class="">
>> ><br class="">
>> ><br class="">
>> > I've got a problem with unicode & xmerl library.<br class="">
>> ><br class="">
>> > Input data for xmerl is utf-8 encoded xml, and what I've got as the result is encoded latin1. But I need utf8!<br class="">
>> ><br class="">
>> ><br class="">
>> > EXAMPLES<br class="">
>> ><br class="">
>> > Body = <<"<?xml version=\"1.0\" encoding=\"UTF-8\"?><response><value>René</value></response>"/utf8>>.<br class="">
>> ><br class="">
>> > (for the sake of portability here is term_to_binary(Body):<br class="">
>> ><br class="">
>> > <<131,109,0,0,0,79,60,63,120,109,108,32,118,101,114,115,<br class="">
>> > 105,111,110,61,34,49,46,48,34,32,101,110,99,111,100,105,<br class="">
>> > 110,103,61,34,85,84,70,45,56,34,63,62,60,114,101,115,<br class="">
>> > 112,111,110,115,101,62,60,118,97,108,117,101,62,82,101,<br class="">
>> > 110,195,169,60,47,118,97,108,117,101,62,60,47,114,101,<br class="">
>> > 115,112,111,110,115,101,62>><br class="">
>> ><br class="">
>> ><br class="">
>> ><br class="">
>> > (1):<br class="">
>> ><br class="">
>> > When I do<br class="">
>> ><br class="">
>> > xmerl_scan:string(binary_to_list(Body)).<br class="">
>> ><br class="">
>> > it returns<br class="">
>> ><br class="">
>> > {#xmlElement{name = response,expanded_name = response,<br class="">
>> > nsinfo = [],<br class="">
>> > namespace = #xmlNamespace{default = [],nodes = []},<br class="">
>> > parents = [],pos = 1,attributes = [],<br class="">
>> > content = [#xmlElement{name = value,expanded_name = value,<br class="">
>> > nsinfo = [],<br class="">
>> > namespace = #xmlNamespace{default = [],nodes = []},<br class="">
>> > parents = [{response,1}],<br class="">
>> > pos = 1,attributes = [],<br class="">
>> > content = [#xmlText{parents = [{value,1},{response,1}],<br class="">
>> > pos = 1,language = [],<br class="">
>> ><br class="">
>> ><br class="">
>> > value = "René",<br class="">
>> ><br class="">
>> ><br class="">
>> > type = text}],<br class="">
>> > language = [],xmlbase = "/Users/aturkin/ws/",<br class="">
>> > elementdef = undeclared}],<br class="">
>> > language = [],xmlbase = "/Users/aturkin/ws/",<br class="">
>> > elementdef = undeclared},<br class="">
>> > []}<br class="">
>> ><br class="">
>> ><br class="">
>> > So, note there is `value = "René"` string, and it uses [233] symbol, which is latin1.<br class="">
>> ><br class="">
>> ><br class="">
>> ><br class="">
>> ><br class="">
>> > (2):<br class="">
>> ><br class="">
>> > xmerl_scan:string(xmerl_ucs:to_utf8(binary_to_list(Body)))<br class="">
>> ><br class="">
>> > returns<br class="">
>> ><br class="">
>> > {#xmlElement{name = response,expanded_name = response,<br class="">
>> > nsinfo = [],<br class="">
>> > namespace = #xmlNamespace{default = [],nodes = []},<br class="">
>> > parents = [],pos = 1,attributes = [],<br class="">
>> > content = [#xmlElement{name = value,expanded_name = value,<br class="">
>> > nsinfo = [],<br class="">
>> > namespace = #xmlNamespace{default = [],nodes = []},<br class="">
>> > parents = [{response,1}],<br class="">
>> > pos = 1,attributes = [],<br class="">
>> > content = [#xmlText{parents = [{value,1},{response,1}],<br class="">
>> > pos = 1,language = [],<br class="">
>> ><br class="">
>> ><br class="">
>> > value = "René",<br class="">
>> ><br class="">
>> ><br class="">
>> > type = text}],<br class="">
>> > language = [],xmlbase = "/Users/aturkin/ws/",<br class="">
>> > elementdef = undeclared}],<br class="">
>> > language = [],xmlbase = "/Users/aturkin/ws/",<br class="">
>> > elementdef = undeclared},<br class="">
>> > []}<br class="">
>> ><br class="">
>> > Now `value = "René"`, so 2 bytes are used to code this symbol, and this is utf-8.<br class="">
>> ><br class="">
>> > So in (2) I get what I need, but why I need to force that conversion for xmerl?<br class="">
>> ><br class="">
>> ><br class="">
>> ><br class="">
>> ><br class="">
>> > QUESTIONS<br class="">
>> ><br class="">
>> > 1. I don't understand why xmerl_scan allows you to set input encoding, but it looks like there is no way to set output encoding. Is there any way to make xmerl_scan to return utf8 instead of latin1?<br class="">
>> ><br class="">
>> > 2. How is that happen, that in (1) it does conversion utf-8 -> latin1, and in (2) it's utf-8?<br class="">
>> ><br class="">
>> ><br class="">
>> ><br class="">
>> ><br class="">
>> > --<br class="">
>> > Best Regards,<br class="">
>> > Alex Turkin<br class="">
>> _______________________________________________<br class="">
>> erlang-questions mailing list<br class="">
>> <a href="mailto:erlang-questions@erlang.org" class="">erlang-questions@erlang.org</a><br class="">
>> <a href="http://erlang.org/mailman/listinfo/erlang-questions" rel="noreferrer" target="_blank" class="">http://erlang.org/mailman/listinfo/erlang-questions</a><br class="">
>><br class="">
>><br class="">
>><br class="">
>><br class="">
>> --<br class="">
>> Best Regards,<br class="">
>> Alex Turkin<br class="">
>> _______________________________________________<br class="">
>> erlang-questions mailing list<br class="">
>> <a href="mailto:erlang-questions@erlang.org" class="">erlang-questions@erlang.org</a><br class="">
>> <a href="http://erlang.org/mailman/listinfo/erlang-questions" rel="noreferrer" target="_blank" class="">http://erlang.org/mailman/listinfo/erlang-questions</a><br class="">
><br class="">
><br class="">
><br class="">
><br class="">
> --<br class="">
> Best Regards,<br class="">
> Alex Turkin<br class="">
<br class="">
</div></div></blockquote></div><br class=""><br clear="all" class=""><div class=""><br class=""></div>-- <br class=""><div class="gmail_signature">Best Regards,<div class="">Alex Turkin</div></div>
</div>
</div></blockquote></div><br class=""></div></div></body></html>