[erlang-questions] Patch Package OTP 22.0.6 Released

[Grzegorz Junka]

On 14/07/2019 15:48, Mikael Pettersson wrote:
> On Fri, Jul 12, 2019 at 12:57 PM Grzegorz Junka <list1@REDACTED> wrote:
>> On 10/07/2019 22:00, Mikael Pettersson wrote:
>>> On Wed, Jul 10, 2019 at 6:08 PM Loïc Hoguin <essen@REDACTED> wrote:
>>>>>      OTP-15949    Application(s): dialyzer, hipe
>>>>>
>>>>>                   The HiPE compiler would badly miscompile certain
>>>>>                   try/catch expressions, so it will now refuse to compile
>>>>>                   modules containing try or catch.
>>>>>
>>>>>                   As a consequence of this, dialyzer will no longer
>>>>>                   compile key modules to native code.
>>>> This means --enable-native-libs is now broken. Results in:
>>>>
>>>> hipe.erl: native-code compilation failed because of an unimplemented
>>>> instruction (catch).
>>>>
>>>> That's a bit rough.
>>>>
>>>> Is HiPE just going to slowly rot or are there plans to revive it somehow
>>> I had this discussion with Björn from the OTP team at the latest EUC
>>> (er, "Code BEAM Sto").
>>>
>>> The issue is that the original implementers of HiPE have long since
>>> moved on to other things
>>> and can't spend time on it any more.  At the same time the OTP team is only able
>>> to support things their customers care about and pay for.  HiPE isn't
>>> one of those things.
>>>
>>> So it's up to the community as a whole to support HiPE, if it wants
>>> it.  This could take the
>>> form of one or more dedicated individuals taking on maintenance of
>>> HiPE and fixing issues
>>> as they are discovered.  (There's at least one such individual working
>>> on the bit syntax
>>> compilation issue in OTP-22 right now.)  Another solution could be for
>>> interested parties to
>>> fund maintenance via the Erlang Ecosystem Foundation.
>>>
>>> The try/catch issue is news to me, but since the bit syntax
>>> compilation issue was known
>>> as OTP-22.0 was released, my view is that OTP should have deprecated
>>> HiPE at that point
>>> with the understanding that OTP-23 would delete HiPE unless the
>>> community stepped up
>>> and fixed the issues.  (And by that I don't mean "silently don't
>>> compile things we can't handle
>>> any more".)
>>>
>>> /Mikael, ex-HiPE developer
>>
>> Mikael, would it be possible for you or someone else from the OTP team
> Note: drop "else", I'm not in the OTP team, just a long-time contributor
> to Erlang/OTP.
>
>> to provide more details about the minimal amount of work required to
>> bring HiPE back to the working order? I am looking for things like:
>>
>> 1. Any test suites available for HiPE (I would expect some tests not
>> passing due to recent changes)
> HiPE used to maintain its own internal test suite.  The HiPE compiler
> application in OTP includes some tests, but I have no idea how
> complete it is; a quick look indicates it doesn't include everything
> from the old internal test suite.  A snapshot of that test suite as of early
> 2011 is available in <https://github.com/andysan/hipe_tests>, but there
> are a few later additions that AFAIK are only available in a 2014 CVS
> snapshot I have locally.
>
>> 2. Documentation detailing the HiPE implementation (if available)
> Go to <https://www.it.uu.se/research/group/hipe/> and read the
> various publications.  Those and the source code (erts/emulator/hipe/,
> other parts of erts/ under #ifdef HIPE, lib/hipe/ (the compiler), and a
> few other bits under lib/) are the documentation.
>
>> 3. New features that the new compiler no longer compiles for HiPE
>> 4. Planned features that will require support in the compiler in the
>> foreseeable future
>>
>> Also, is the HiPE code tightly integrated with OTP and its compiler or
>> it could be extracted into a separate repo and maintained at its own peace?
> HiPE and OTP are mutually dependent:
> - the HiPE compiler's input is the BEAM code output by the OTP compiler,
>    so whenever BEAM instructions are added or the OTP compiler changes
>    drastically, the HiPE compiler needs to be updated
> - the OTP compiler dispatches to the HiPE compiler if you compile with the
>    native flag enabled
> - the OTP code loader dispatches to the HiPE code loader for modules
>    compiled to native code
> - the OTP VM (aka runtime system) has extensions to
>    + load native code
>    + allow Erlang processes to call native code from BEAM and vice versa
>    + likewise handle throwing exceptions across the two execution modes
>    + garbage collection of the native code's stack
>    + handling dynamic (re-)loading of modules containing native code


Sorry, forgot to ask one more question. If HiPE compiler's input is the 
BEAM code then I understand interested in HiPE may not only be the 
Erlang community but also other communities that use the BEAM VM, e.g. 
Elixir and some other listed here 
https://github.com/llaisdy/beam_languages ?

GrzegorzJ