[erlang-questions] Building / releasing with HIPE / native

[Adrian Roe]

Paul / Kostis / Group  

First off many thanks for your help - I have got to the bottom of what was happening.  Basically there are beam files included in the src release tar balls that meant certain beams were not being built.  The details are below and I would be interested to know if the presence of the beam files in the source tar balls is deliberate.

I was building Erlang from a script that did the following:

# Download and install erlang
OtpDir=otp_src_R16B03-1

rm -rf $OtpDir

wget http://www.erlang.org/download/otp_src_R16B03-1.tar.gz
tar xf otp_src_R16B03-1.tar.gz
rm otp_src_R16B03-1.tar.gz

pushd $OtpDir

./configure --enable-native-libs
make
sudo make install
popd



Every time it tried again I would re-run the script so that what I was doing was entirely reproducible.

The issue is that the otp_src_R16B03-1.tar.gz contains files in otp_src_R16B03-1/lib/stdlib/ebin/ including lists.beam and others and the make process above was not marking the beams in that directory as obsolete (the beam files don't have a dependency on the Makefile).  I was concerned about side-effects from previous attempts impacting settings so between every attempt ran "rm -rf otp_src_R16B03-1” and started again - reintroducing the obsolete beams every time.

I think the minimal set of commands to get a native build from the src release tar is:

./configure --enable-native-libs
make clean # you need to run ./configure before clean as you don’t have a Makefile otherwise
./configure --enable-native-libs # as the make clean will have deleted the settings
make
sudo make install



This deletes the beams included in the top tar ball and then runs make from a clean base.  Pulling the source from Github should work as well as the beams are not part of the repo.   

Thanks again for your help,

Adrian
--  
Adrian Roe
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)

On Wednesday, 2 April 2014 at 21:31, Kostis Sagonas wrote:  
> On 04/02/2014 08:11 PM, Adrian Roe wrote:
> > We’ve been using Erlang in production for quite a few years now and for
> > the first time we have a project that really benefits from HIPE / native
> > code being turned on (before things have always been fast enough without
> > and the benefit of more detailed stack traces etc outweighed any
> > performance gains). We did some benchmarking by compiling single
> > modules with HIPE - both in our application code and then going through
> > key OTP / Erlang library modules, clearing sticky_directory errors and
> > recompiling them one by one with +native.
> >  
> > Now that it is clear that (for this use case) native / HIPE is the way
> > to go, I went to the project’s build box and recompiled Erlang (R16B03)
> > from source with
> >  
> > ./configure --enable-hipe --enable-native-libs
> > in the expectation that all libraries would be compiled with +native
> >  
> > I’ve tried this on OSX, CentOS and Ubuntu all with the same result, so
> > it’s clearly me doing something stupid, but I can’t see it.
> >  
>  
>  
> Can it be that you have not done a 'make clean'? Note that ./configure  
> (with or without flags) does not actually build the system or recreate  
> any dependencies.
>  
> Can you try the following sequence of commands on your Ubuntu machine  
> (unfortunately, I've never tried HiPE on OSX or CentOS to know what is  
> its status there) and tell us what happened?
>  
> make clean
> ./otp_build autoconf
> ./configure --enable-native-libs
> make -j
>  
> As Sverker suggested, you can find out whether some module, say lists,  
> contains native code or not by: code:is_module_native(lists).
>  
>  
> Some more info:
>  
> - Note that for better or worse, --enable-native-libs does not build  
> the complete set of libraries with native code. Only kernel, stdlib,  
> compiler, hipe, syntax_tools and dialyzer are built with native code. If  
> your application requires some other modules in native code, you need to  
> explicitly compile them to native or change the Makefile of their  
> library in a way similar to the Makefiles of these libraries.
>  
> - I would recommend against R16B03 as it is seriously broken in some  
> respects. (This comment is unrelated to native code.) Use R16B03-1  
> instead.
>  
> Kostis  

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20140403/605dd13e/attachment.htm>