[erlang-questions] Different behavior between erl and escript when the string module is used for unicode strings

[Kawatake Chiharu]

Hi,

I found that erl and escript behaved differently when the string module was
used for unicode strings.

Attached are what I tried to run with erl and escript. I also copied the
code at the bottom of this e-mail.

When I did

%> escript test.erl

This works fine.

However, when I used erl to run it, I got an error as follows.

------------------
%> erlc test.erl
%> erl -noshell -s test main -s init stop
{"init terminating in
do_boot",{undef,[{test,main,[]},{init,start_it,1},{init,start_em,1}]}}

Crash dump was written to: erl_crash.dump
init terminating in do_boot ()
-----------------

After looking into this for a while, I start thinking that it seems that
this occurs when I use string:tokens (Or some function in the string module.
I have not tried other functions in the string module yet.)

I am wondering what difference between erl and escript causes this.
Is there anyone who have had a similar thing?

Chiharu


------------------------

-module(test).

-compile(export_all).

for_each_line(Filename, [_, {encoding, Encoding}]=Mode, F, Args) ->
    io:setopts([{encoding, Encoding}]),
    case file:open(Filename, Mode) of
        {ok, Device} ->
            F(Device, Encoding, Args);
        {error, Reason} ->
            erlang:error(Reason)
    end.

tokens(Device, Encoding, N) ->
    case io:get_line(Device, "") of
        {error, Reason} ->
            erlang:error(Reason),
            file:close(Device);
        eof ->
            file:close(Device);
        Line ->
            Tokens = string:tokens(Line, " "),
            lists:foreach(
              fun(T) -> io:format("~ts~n", [T]) end,
              Tokens
             ),
            tokens(Device, Encoding, N)
    end.

main(_) ->
    for_each_line("test1.txt", [read, {encoding, utf8}], fun test:tokens/3,
1).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20090501/dc6de82f/attachment.htm>
-------------- next part --------------
hoge,
hoge hoge,
hoge hoge hoge,
hoge hoge hoge,
hoge,
hoge,
hoge hoge,
hoge,
hoge,
hoge,
hoge hoge hoge hoge,
hoge,
hoge hoge,
hoge hoge,
hoge hoge,
hoge hoge hoge hoge,
hoge hoge hoge hoge,
hoge hoge,
hoge hoge hoge hoge,
hoge hoge hoge,
hoge,
hoge hoge,
hoge,
hoge hoge hoge hoge,
hoge hoge hoge hoge,
hoge hoge hoge,
hoge hoge,
hoge,
hoge hoge hoge,
hoge hoge,
hoge hoge,
hoge hoge,
hoge hoge hoge,
hoge hoge,
hoge,
hoge hoge,
hoge hoge,
hoge,
hoge,
hoge hoge hoge hoge,
hoge hoge hoge hoge,
hoge hoge hoge,
hoge hoge hoge hoge,
hoge,
hoge hoge,
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test.erl
Type: application/octet-stream
Size: 850 bytes
Desc: not available
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20090501/dc6de82f/attachment.obj>