[erlang-questions] Patch Package OTP 22.0.6 Released

[Mikael Pettersson]

On Mon, Jul 15, 2019 at 2:00 PM Grzegorz Junka <list1@REDACTED> wrote:
>
>
> On 14/07/2019 15:48, Mikael Pettersson wrote:
> > On Fri, Jul 12, 2019 at 12:57 PM Grzegorz Junka <list1@REDACTED> wrote:
> >> On 10/07/2019 22:00, Mikael Pettersson wrote:
> >>> On Wed, Jul 10, 2019 at 6:08 PM Loïc Hoguin <essen@REDACTED> wrote:
> >>>>>      OTP-15949    Application(s): dialyzer, hipe
> >>>>>
> >>>>>                   The HiPE compiler would badly miscompile certain
> >>>>>                   try/catch expressions, so it will now refuse to compile
> >>>>>                   modules containing try or catch.
> >>>>>
> >>>>>                   As a consequence of this, dialyzer will no longer
> >>>>>                   compile key modules to native code.
> >>>> This means --enable-native-libs is now broken. Results in:
> >>>>
> >>>> hipe.erl: native-code compilation failed because of an unimplemented
> >>>> instruction (catch).
> >>>>
> >>>> That's a bit rough.
> >>>>
> >>>> Is HiPE just going to slowly rot or are there plans to revive it somehow
> >>> I had this discussion with Björn from the OTP team at the latest EUC
> >>> (er, "Code BEAM Sto").
> >>>
> >>> The issue is that the original implementers of HiPE have long since
> >>> moved on to other things
> >>> and can't spend time on it any more.  At the same time the OTP team is only able
> >>> to support things their customers care about and pay for.  HiPE isn't
> >>> one of those things.
> >>>
> >>> So it's up to the community as a whole to support HiPE, if it wants
> >>> it.  This could take the
> >>> form of one or more dedicated individuals taking on maintenance of
> >>> HiPE and fixing issues
> >>> as they are discovered.  (There's at least one such individual working
> >>> on the bit syntax
> >>> compilation issue in OTP-22 right now.)  Another solution could be for
> >>> interested parties to
> >>> fund maintenance via the Erlang Ecosystem Foundation.
> >>>
> >>> The try/catch issue is news to me, but since the bit syntax
> >>> compilation issue was known
> >>> as OTP-22.0 was released, my view is that OTP should have deprecated
> >>> HiPE at that point
> >>> with the understanding that OTP-23 would delete HiPE unless the
> >>> community stepped up
> >>> and fixed the issues.  (And by that I don't mean "silently don't
> >>> compile things we can't handle
> >>> any more".)
> >>>
> >>> /Mikael, ex-HiPE developer
> >>
> >> Mikael, would it be possible for you or someone else from the OTP team
> > Note: drop "else", I'm not in the OTP team, just a long-time contributor
> > to Erlang/OTP.
> >
> >> to provide more details about the minimal amount of work required to
> >> bring HiPE back to the working order? I am looking for things like:
> >>
> >> 1. Any test suites available for HiPE (I would expect some tests not
> >> passing due to recent changes)
> > HiPE used to maintain its own internal test suite.  The HiPE compiler
> > application in OTP includes some tests, but I have no idea how
> > complete it is; a quick look indicates it doesn't include everything
> > from the old internal test suite.  A snapshot of that test suite as of early
> > 2011 is available in <https://github.com/andysan/hipe_tests>, but there
> > are a few later additions that AFAIK are only available in a 2014 CVS
> > snapshot I have locally.
> >
> >> 2. Documentation detailing the HiPE implementation (if available)
> > Go to <https://www.it.uu.se/research/group/hipe/> and read the
> > various publications.  Those and the source code (erts/emulator/hipe/,
> > other parts of erts/ under #ifdef HIPE, lib/hipe/ (the compiler), and a
> > few other bits under lib/) are the documentation.
> >
> >> 3. New features that the new compiler no longer compiles for HiPE
> >> 4. Planned features that will require support in the compiler in the
> >> foreseeable future
> >>
> >> Also, is the HiPE code tightly integrated with OTP and its compiler or
> >> it could be extracted into a separate repo and maintained at its own peace?
> > HiPE and OTP are mutually dependent:
> > - the HiPE compiler's input is the BEAM code output by the OTP compiler,
> >    so whenever BEAM instructions are added or the OTP compiler changes
> >    drastically, the HiPE compiler needs to be updated
> > - the OTP compiler dispatches to the HiPE compiler if you compile with the
> >    native flag enabled
> > - the OTP code loader dispatches to the HiPE code loader for modules
> >    compiled to native code
> > - the OTP VM (aka runtime system) has extensions to
> >    + load native code
> >    + allow Erlang processes to call native code from BEAM and vice versa
> >    + likewise handle throwing exceptions across the two execution modes
> >    + garbage collection of the native code's stack
> >    + handling dynamic (re-)loading of modules containing native code
>
>
> Sorry, forgot to ask one more question. If HiPE compiler's input is the
> BEAM code then I understand interested in HiPE may not only be the
> Erlang community but also other communities that use the BEAM VM, e.g.
> Elixir and some other listed here
> https://github.com/llaisdy/beam_languages ?

HiPE should work with any language that compiles to Erlang.
It may also work with languages that compile via Core Erlang, but
that path is less tested.