[erlang-questions] Patch Package OTP 22.0.6 Released

[Mikael Pettersson]

On Fri, Jul 12, 2019 at 12:57 PM Grzegorz Junka <list1@REDACTED> wrote:
>
> On 10/07/2019 22:00, Mikael Pettersson wrote:
> > On Wed, Jul 10, 2019 at 6:08 PM Loïc Hoguin <essen@REDACTED> wrote:
> >>>     OTP-15949    Application(s): dialyzer, hipe
> >>>
> >>>                  The HiPE compiler would badly miscompile certain
> >>>                  try/catch expressions, so it will now refuse to compile
> >>>                  modules containing try or catch.
> >>>
> >>>                  As a consequence of this, dialyzer will no longer
> >>>                  compile key modules to native code.
> >> This means --enable-native-libs is now broken. Results in:
> >>
> >> hipe.erl: native-code compilation failed because of an unimplemented
> >> instruction (catch).
> >>
> >> That's a bit rough.
> >>
> >> Is HiPE just going to slowly rot or are there plans to revive it somehow
> > I had this discussion with Björn from the OTP team at the latest EUC
> > (er, "Code BEAM Sto").
> >
> > The issue is that the original implementers of HiPE have long since
> > moved on to other things
> > and can't spend time on it any more.  At the same time the OTP team is only able
> > to support things their customers care about and pay for.  HiPE isn't
> > one of those things.
> >
> > So it's up to the community as a whole to support HiPE, if it wants
> > it.  This could take the
> > form of one or more dedicated individuals taking on maintenance of
> > HiPE and fixing issues
> > as they are discovered.  (There's at least one such individual working
> > on the bit syntax
> > compilation issue in OTP-22 right now.)  Another solution could be for
> > interested parties to
> > fund maintenance via the Erlang Ecosystem Foundation.
> >
> > The try/catch issue is news to me, but since the bit syntax
> > compilation issue was known
> > as OTP-22.0 was released, my view is that OTP should have deprecated
> > HiPE at that point
> > with the understanding that OTP-23 would delete HiPE unless the
> > community stepped up
> > and fixed the issues.  (And by that I don't mean "silently don't
> > compile things we can't handle
> > any more".)
> >
> > /Mikael, ex-HiPE developer
>
>
> Mikael, would it be possible for you or someone else from the OTP team

Note: drop "else", I'm not in the OTP team, just a long-time contributor
to Erlang/OTP.

> to provide more details about the minimal amount of work required to
> bring HiPE back to the working order? I am looking for things like:
>
> 1. Any test suites available for HiPE (I would expect some tests not
> passing due to recent changes)

HiPE used to maintain its own internal test suite.  The HiPE compiler
application in OTP includes some tests, but I have no idea how
complete it is; a quick look indicates it doesn't include everything
from the old internal test suite.  A snapshot of that test suite as of early
2011 is available in <https://github.com/andysan/hipe_tests>, but there
are a few later additions that AFAIK are only available in a 2014 CVS
snapshot I have locally.

> 2. Documentation detailing the HiPE implementation (if available)

Go to <https://www.it.uu.se/research/group/hipe/> and read the
various publications.  Those and the source code (erts/emulator/hipe/,
other parts of erts/ under #ifdef HIPE, lib/hipe/ (the compiler), and a
few other bits under lib/) are the documentation.

> 3. New features that the new compiler no longer compiles for HiPE
> 4. Planned features that will require support in the compiler in the
> foreseeable future
>
> Also, is the HiPE code tightly integrated with OTP and its compiler or
> it could be extracted into a separate repo and maintained at its own peace?

HiPE and OTP are mutually dependent:
- the HiPE compiler's input is the BEAM code output by the OTP compiler,
  so whenever BEAM instructions are added or the OTP compiler changes
  drastically, the HiPE compiler needs to be updated
- the OTP compiler dispatches to the HiPE compiler if you compile with the
  native flag enabled
- the OTP code loader dispatches to the HiPE code loader for modules
  compiled to native code
- the OTP VM (aka runtime system) has extensions to
  + load native code
  + allow Erlang processes to call native code from BEAM and vice versa
  + likewise handle throwing exceptions across the two execution modes
  + garbage collection of the native code's stack
  + handling dynamic (re-)loading of modules containing native code

You can't distribute this as just another "application".

As for the concrete problems right now:
1. The binary compilation issue is documented in both the bug tracker
    and as a PR on github, so I won't expand on that.
2. The "try/catch miscompilation" issue is mentioned in the OTP-22.0.6
    release notes, but the only reference is to an OTP-internal bug entry
    (OTP-15949) and there's no test case added, making it difficult to
    understand what the issue is or how to fix it.

Finally, a major part of the problem is that OTP is a moving target.  The
VM keeps changing (new BEAM instructions, new conventions for
memory management or BIF calls), and the OTP compiler also changes,
and each such change has the potential for breaking HiPE.

/Mikael