[erlang-questions] SOLVED -- Re: Dangers of generating a large erlang module

Ivan Uemlianin <>
Mon Sep 30 10:09:16 CEST 2013


Dear All

Thanks to everyone for all help and comments.

A less glamorous solution than Core Erlang came to me last night while I 
was doing the ironing.

As binaries seemed to be the problem I worked out a way to remove all 
binaries from the generated code, and use lists instead.  Then I have a 
wrapper function in the non-generated (?) code which converts binaries 
to lists and lists to binaries as appropriate.

Now the 120k+ ruleset compiles in a few minutes on a 32bit emulator, so 
it doesn't look like there's any danger of the running out of memory.

Using the ruleset is still amazingly fast: less than ten *micro*seconds 
for the wrapper function to return.

I'll still look into Core Erlang so I'm ready for next time.

Very best wishes to all

Ivan


On 30/09/2013 06:31, Erik Søe Sørensen wrote:
> Yes, indeed; considering that the goal is reduction of memory use,
> however, what we want is to avoid intermediate representations which
> explode in size on binaries. I don't know anything about the from_core
> parser, but the string-as-list representation of a binary literal is in
> itself a factor 32/64 blowup. :-/
> (Btw: I was wrong about introductions to Core Erlang; threre is one.
> Writing on the phone has its fact-checking disadvantages...)
>
> Den 29/09/2013 23.01 skrev "Tom Murphy" <
> <mailto:>>:
>
>     There's also just:
>
>     erlc +to_core foo.erl
>     erlc +from_core foo.core
>
>     Tom Murphy
>
>
>     On Sun, Sep 29, 2013 at 11:59 AM, Loïc Hoguin <
>     <mailto:>> wrote:
>
>         On 09/29/2013 08:24 PM, Ivan uemlianin wrote:
>
>             That's what I've just done :D  Core Erlang looks very
>             verbose but quite
>             regular & probably not difficult to generate.
>
>             My questions now are:
>             - are there any libraries "out there" for generating Core
>             Erlang, or do
>             we all roll our own?
>
>
>         Look at the cerl module. It's just a matter of generating the
>         proper structure using those functions.
>
>
>             - how would one use compile:file or compile:forms with core
>             erlang? I
>             haven't been able to find any documentation (haven't read
>             Richard
>             Carlsson's Introduction paper yet).
>
>
>         Do read it. But to answer that question, compile:forms with
>         from_core option.
>
>             Many thanks
>
>             Ivan
>
>             --
>             festina lente
>
>
>             On 29 Sep 2013, at 18:36, Erik Søe Sørensen
>             < <mailto:>
>             <mailto: <mailto:>>> wrote:
>
>                 Core Erlang is an intermediate  representation in the
>                 Erlang compiler
>                 - but also (afaik) a fairly well-defined/public one and
>                 one that is
>                 stable.
>                 I don't think you'll find much in the vein of tutorials.
>                 Try getting
>                 erlc to output the intermediate format, though, for a
>                 small program
>                 similar to what you'll be using it for.
>
>                 Den 29/09/2013 19.20 skrev "Ivan uemlianin"
>                 < <mailto:>
>                 <mailto: <mailto:>>>:
>
>
>                      Thanks! I think I'll try and head in that
>                 direction. I've had a
>                      few goes at other methods (db lookup etc) and
>                 they're much slower
>                      than this "dynamic hardcoding"). I'll sniff around
>                 for Core Erlang
>                      tutorials.
>
>                      Best wishes
>
>                      Ivan
>
>
>                      --
>                      festina lente
>
>
>                      On 29 Sep 2013, at 17:48, Erik Søe Sørensen
>                 < <mailto:>
>                      <mailto:
>                 <mailto:>>> wrote:
>
>                          A thing which I discovered recently (in
>                     connection with
>                          mochiglobal) is that compiling code containing
>                     large binaries, or
>                          large amounts of binaries,  is quite
>                     memory-intensive. As I
>                          recall it, the numbers were ~64 bytes of RAM
>                     per byte in a binary
>                          metal; twice as much if on a 64 bit emulator.
>                          Which means that if you want to compile modules
>                     containing (in
>                          sum) multimegabyte binaries, doing so from
>                     Erlang source or from
>                          full Erlang AST is a no-go.  Iirc, it is
>                     feasible if starting
>                          from Core Erlang.
>                          /Erik
>
>                          Den 29/09/2013 12.50 skrev "Ivan Uemlianin"
>                     < <mailto:>
>                          <mailto:
>                     <mailto:>>>:
>
>
>                              Dear Anthony
>
>                              Thanks for your comment.
>
>                              Yes, that's exactly what the generated
>                     module is doing.  The
>                              generated module has a single function with
>                     many clauses like
>                              this:
>
>                                  f(<<"trigger", Rest/binary) -> ...
>
>                              This is why (as far as I can work out) the
>                     generated code has
>                              to be so big.
>
>                              I prefer the idea of generating and loading
>                     code to, say,
>                              updating a database table, because it seems
>                     faster and less
>                              likely to lead to bottlenecks.
>
>                              Best wishes
>
>                              Ivan
>
>
>                              On 29/09/2013 11:38, Anthony Ramine wrote:
>
>                                  Hello Ivan,
>
>                                  Out of curiosity, what does it look like?
>
>                                  Pattern matching on literal values in
>                     Erlang is done with
>                                  a binary search over the sorted list of
>                     patterns, I am
>                                  not sure this would play well with your
>                     use case even if
>                                  the compilation didn't bring the VM down.
>
>                                  Regards,
>
>                                  Le 29 sept. 2013 à 11:29, Ivan
>                     Uemlianin a écrit :
>
>                                      All goes well on small test files,
>                     but the files I
>                                      want to use IRL are relatively
>                     large --- around
>                                      120,000 lines.
>
>
>
>                              --
>
>                     ==============================____============================__==
>
>                              Ivan A. Uemlianin PhD
>                              Llaisdy
>                              Speech Technology Research and Development
>
>                      <mailto:>
>                     <mailto: <mailto:>>
>                     www.llaisdy.com <http://www.llaisdy.com>
>                     <http://www.llaisdy.com>
>                     llaisdy.wordpress.com <http://llaisdy.wordpress.com>
>                     <http://llaisdy.wordpress.com>
>                     github.com/llaisdy <http://github.com/llaisdy>
>                     <http://github.com/llaisdy>
>                     www.linkedin.com/in/____ivanuemlianin
>                     <http://www.linkedin.com/in/__ivanuemlianin>
>                              <http://www.linkedin.com/in/__ivanuemlianin
>                     <http://www.linkedin.com/in/ivanuemlianin>>
>
>                                                      festina lente
>
>                     ==============================____============================__==
>
>                     ___________________________________________________
>                              erlang-questions mailing list
>                     
>                     <mailto:>
>                     <mailto:
>                     <mailto:>>
>                     http://erlang.org/mailman/____listinfo/erlang-questions
>                     <http://erlang.org/mailman/__listinfo/erlang-questions>
>
>                     <http://erlang.org/mailman/__listinfo/erlang-questions
>                     <http://erlang.org/mailman/listinfo/erlang-questions>>
>
>
>
>             _________________________________________________
>             erlang-questions mailing list
>              <mailto:>
>             http://erlang.org/mailman/__listinfo/erlang-questions
>             <http://erlang.org/mailman/listinfo/erlang-questions>
>
>
>
>         --
>         Loïc Hoguin
>         Erlang Cowboy
>         Nine Nines
>         http://ninenines.eu
>
>         _________________________________________________
>         erlang-questions mailing list
>          <mailto:>
>         http://erlang.org/mailman/__listinfo/erlang-questions
>         <http://erlang.org/mailman/listinfo/erlang-questions>
>
>
>
>     _______________________________________________
>     erlang-questions mailing list
>      <mailto:>
>     http://erlang.org/mailman/listinfo/erlang-questions
>
>
>
> _______________________________________________
> erlang-questions mailing list
> 
> http://erlang.org/mailman/listinfo/erlang-questions
>

-- 
============================================================
Ivan A. Uemlianin PhD
Llaisdy
Speech Technology Research and Development

                     
                      www.llaisdy.com
                          llaisdy.wordpress.com
               github.com/llaisdy
                      www.linkedin.com/in/ivanuemlianin

                         festina lente
============================================================


More information about the erlang-questions mailing list