[erlang-questions] HTML Entity Refs and xmerl

[Caoyuan]

According to http://www.erlang.org/pipermail/erlang-bugs/2007-May/000345.html,
currently xmerl_scan only recognizes the very limited set of entity
references. In brief, if you try to xmerl:scan xml text that includes
standard HTML entity refs, such as nbsp, iexcl, pound, frac14, etc.
you'll encounter something like:

16> edoc:file("exprecs.erl").
2670- fatal: {unknown_entity_ref,nbsp}
2580- fatal: error_scanning_entity_ref
exprecs.erl, in module header: at line 28: error in XML parser: {fatal,
                         {error_scanning_entity_ref,
                             {file,file_name_unknown},
                             {line,86},
                             {col,18}}}.
** exited: error **

Ulf Wiger said:
... I realize that xmerl can be customized with a rules function
which, for example, can handle entity references...

So I take a try by writing a piece of code (html_entity_refs.erl) to
parse a HTML entity ref DTD file to ets rules, then:

xmerl_scan:string(XmlText, [{rules, html_entity_refs:get_xmerl_rules()}]).

Yes, this works. But for a 3MB testing file, the parsing took about 30 seconds.

How about convert these entity refs to utf-8 chars first, then apply
xmerl_scan to it?
I wrote another piece of code, and now:

xmerl_scan:string(html_entity_refs:decode_for_xml(XmlText)).

This time, the decoding+parsing time is about 5 seconds, it's 6 times
faster than ets rules solution.

The html_entity_refs.erl can be got from:
http://caoyuan.googlecode.com/svn/trunk/erlang/html_entity_refs.erl