[erlang-questions] SOLVED -- Re: Dangers of generating a large erlang module
Ivan Uemlianin
ivan@REDACTED
Mon Sep 30 10:09:16 CEST 2013
Dear All
Thanks to everyone for all help and comments.
A less glamorous solution than Core Erlang came to me last night while I
was doing the ironing.
As binaries seemed to be the problem I worked out a way to remove all
binaries from the generated code, and use lists instead. Then I have a
wrapper function in the non-generated (?) code which converts binaries
to lists and lists to binaries as appropriate.
Now the 120k+ ruleset compiles in a few minutes on a 32bit emulator, so
it doesn't look like there's any danger of the running out of memory.
Using the ruleset is still amazingly fast: less than ten *micro*seconds
for the wrapper function to return.
I'll still look into Core Erlang so I'm ready for next time.
Very best wishes to all
Ivan
On 30/09/2013 06:31, Erik Søe Sørensen wrote:
> Yes, indeed; considering that the goal is reduction of memory use,
> however, what we want is to avoid intermediate representations which
> explode in size on binaries. I don't know anything about the from_core
> parser, but the string-as-list representation of a binary literal is in
> itself a factor 32/64 blowup. :-/
> (Btw: I was wrong about introductions to Core Erlang; threre is one.
> Writing on the phone has its fact-checking disadvantages...)
>
> Den 29/09/2013 23.01 skrev "Tom Murphy" <amindfv@REDACTED
> <mailto:amindfv@REDACTED>>:
>
> There's also just:
>
> erlc +to_core foo.erl
> erlc +from_core foo.core
>
> Tom Murphy
>
>
> On Sun, Sep 29, 2013 at 11:59 AM, Loïc Hoguin <essen@REDACTED
> <mailto:essen@REDACTED>> wrote:
>
> On 09/29/2013 08:24 PM, Ivan uemlianin wrote:
>
> That's what I've just done :D Core Erlang looks very
> verbose but quite
> regular & probably not difficult to generate.
>
> My questions now are:
> - are there any libraries "out there" for generating Core
> Erlang, or do
> we all roll our own?
>
>
> Look at the cerl module. It's just a matter of generating the
> proper structure using those functions.
>
>
> - how would one use compile:file or compile:forms with core
> erlang? I
> haven't been able to find any documentation (haven't read
> Richard
> Carlsson's Introduction paper yet).
>
>
> Do read it. But to answer that question, compile:forms with
> from_core option.
>
> Many thanks
>
> Ivan
>
> --
> festina lente
>
>
> On 29 Sep 2013, at 18:36, Erik Søe Sørensen
> <eriksoe@REDACTED <mailto:eriksoe@REDACTED>
> <mailto:eriksoe@REDACTED <mailto:eriksoe@REDACTED>>> wrote:
>
> Core Erlang is an intermediate representation in the
> Erlang compiler
> - but also (afaik) a fairly well-defined/public one and
> one that is
> stable.
> I don't think you'll find much in the vein of tutorials.
> Try getting
> erlc to output the intermediate format, though, for a
> small program
> similar to what you'll be using it for.
>
> Den 29/09/2013 19.20 skrev "Ivan uemlianin"
> <ivan@REDACTED <mailto:ivan@REDACTED>
> <mailto:ivan@REDACTED <mailto:ivan@REDACTED>>>:
>
>
> Thanks! I think I'll try and head in that
> direction. I've had a
> few goes at other methods (db lookup etc) and
> they're much slower
> than this "dynamic hardcoding"). I'll sniff around
> for Core Erlang
> tutorials.
>
> Best wishes
>
> Ivan
>
>
> --
> festina lente
>
>
> On 29 Sep 2013, at 17:48, Erik Søe Sørensen
> <eriksoe@REDACTED <mailto:eriksoe@REDACTED>
> <mailto:eriksoe@REDACTED
> <mailto:eriksoe@REDACTED>>> wrote:
>
> A thing which I discovered recently (in
> connection with
> mochiglobal) is that compiling code containing
> large binaries, or
> large amounts of binaries, is quite
> memory-intensive. As I
> recall it, the numbers were ~64 bytes of RAM
> per byte in a binary
> metal; twice as much if on a 64 bit emulator.
> Which means that if you want to compile modules
> containing (in
> sum) multimegabyte binaries, doing so from
> Erlang source or from
> full Erlang AST is a no-go. Iirc, it is
> feasible if starting
> from Core Erlang.
> /Erik
>
> Den 29/09/2013 12.50 skrev "Ivan Uemlianin"
> <ivan@REDACTED <mailto:ivan@REDACTED>
> <mailto:ivan@REDACTED
> <mailto:ivan@REDACTED>>>:
>
>
> Dear Anthony
>
> Thanks for your comment.
>
> Yes, that's exactly what the generated
> module is doing. The
> generated module has a single function with
> many clauses like
> this:
>
> f(<<"trigger", Rest/binary) -> ...
>
> This is why (as far as I can work out) the
> generated code has
> to be so big.
>
> I prefer the idea of generating and loading
> code to, say,
> updating a database table, because it seems
> faster and less
> likely to lead to bottlenecks.
>
> Best wishes
>
> Ivan
>
>
> On 29/09/2013 11:38, Anthony Ramine wrote:
>
> Hello Ivan,
>
> Out of curiosity, what does it look like?
>
> Pattern matching on literal values in
> Erlang is done with
> a binary search over the sorted list of
> patterns, I am
> not sure this would play well with your
> use case even if
> the compilation didn't bring the VM down.
>
> Regards,
>
> Le 29 sept. 2013 à 11:29, Ivan
> Uemlianin a écrit :
>
> All goes well on small test files,
> but the files I
> want to use IRL are relatively
> large --- around
> 120,000 lines.
>
>
>
> --
>
> ==============================____============================__==
>
> Ivan A. Uemlianin PhD
> Llaisdy
> Speech Technology Research and Development
>
> ivan@REDACTED <mailto:ivan@REDACTED>
> <mailto:ivan@REDACTED <mailto:ivan@REDACTED>>
> www.llaisdy.com <http://www.llaisdy.com>
> <http://www.llaisdy.com>
> llaisdy.wordpress.com <http://llaisdy.wordpress.com>
> <http://llaisdy.wordpress.com>
> github.com/llaisdy <http://github.com/llaisdy>
> <http://github.com/llaisdy>
> www.linkedin.com/in/____ivanuemlianin
> <http://www.linkedin.com/in/__ivanuemlianin>
> <http://www.linkedin.com/in/__ivanuemlianin
> <http://www.linkedin.com/in/ivanuemlianin>>
>
> festina lente
>
> ==============================____============================__==
>
> ___________________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> <mailto:erlang-questions@REDACTED>
> <mailto:erlang-questions@REDACTED
> <mailto:erlang-questions@REDACTED>>
> http://erlang.org/mailman/____listinfo/erlang-questions
> <http://erlang.org/mailman/__listinfo/erlang-questions>
>
> <http://erlang.org/mailman/__listinfo/erlang-questions
> <http://erlang.org/mailman/listinfo/erlang-questions>>
>
>
>
> _________________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED <mailto:erlang-questions@REDACTED>
> http://erlang.org/mailman/__listinfo/erlang-questions
> <http://erlang.org/mailman/listinfo/erlang-questions>
>
>
>
> --
> Loïc Hoguin
> Erlang Cowboy
> Nine Nines
> http://ninenines.eu
>
> _________________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED <mailto:erlang-questions@REDACTED>
> http://erlang.org/mailman/__listinfo/erlang-questions
> <http://erlang.org/mailman/listinfo/erlang-questions>
>
>
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED <mailto:erlang-questions@REDACTED>
> http://erlang.org/mailman/listinfo/erlang-questions
>
>
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
--
============================================================
Ivan A. Uemlianin PhD
Llaisdy
Speech Technology Research and Development
ivan@REDACTED
www.llaisdy.com
llaisdy.wordpress.com
github.com/llaisdy
www.linkedin.com/in/ivanuemlianin
festina lente
============================================================
More information about the erlang-questions
mailing list