HiPE to be removed in OTP 24?

[Grzegorz Junka]

On 21/06/2020 10:16, Mikael Pettersson wrote:
> On Sat, Jun 20, 2020 at 10:05 PM Grzegorz Junka <list1@REDACTED> wrote:
>> OK, there is a bunch of limitations listed in http://erlang.org/doc/man/HiPE_app.html#feature-limitations
>>
>> Erlang tracing is one of them. Are those limitations inherent to the HiPE compiler, i.e. they can't be fixed,
>> or they are simply there because they haven't been implemented? Or in other words, is AOT compilation
>> of Erlang code into native code doomed and JIT is the only solution to speed it up?
> AOT compilation of Erlang is not inherently difficult.  The problems
> faced by HiPE are due to a few
> unfortunate facts:
>
> ...
>
> A way to resurrect HiPE could be to:
>
> A. Retarget the compiler to consume a higher-level more stable
> intermediate representation from the
>      BEAM compiler, perhaps Core Erlang but even that is a moving target.
> B. Drop all remnants of being able to native-compile individual MFAs,
> just handle whole modules.
>       (That restriction is already in place, but the internals still
> operate on individual MFAs.)
> C. Fix the bugs, streamline the VM interfaces between BEAM and HiPE.
>
> Realistically this would require 6 person-months or more.


Thanks Mikael for the explanation. I understand the facts but still 
don't understand why it's worth investing in JIT instead of developing 
the features you listed that would be needed to resurrect HiPE.

For example, you mentioned HiPE was abandoned 10 years ago, but that's 
roughly also the time when the core team started working on the first 
implementations of JIT. Did you think at that time that JIT would be 
less effort than improving HiPE, or maybe there was no personal interest 
in HiPE among the core team?

To me, as a software developer, it just seems strange to abandon HiPE 
and allow the code to deteriorate after it has been proven to work, and 
instead put so much effort into JIT that has never worked and never been 
proven to work as well as HiPE.

Maybe there are some politics involved between the core team and the 
team that worked on HiPE that you don't want to mention, which is fine, 
or maybe there were some mistakes made in the past that no one wants to 
admit to, which is again understandable. But none of the technical 
reasons I've read so far favor JIT over HiPE, so I hope you don't mind 
me drilling this subject.

One technical reason mentioned, that wasn't related to the abandonment 
of the code, was that JIT would benefit both, Elixir and Erlang. But 
surely that must be also true if HiPE targets the intermediate 
representation from the BEAM compiler?

Grzegorz