HiPE to be removed in OTP 24?

[Valentin Micic]

First, my apologies for accidentally sending an empty message (my fingers were moving whilst my brain was farting).

I am a late adopter of HiPE — even though I did some testing over years (and, mind you,  it was fairly shaky in early days, hence my reluctance to use it in production), it took years before I really found a good reason (or became desperate enough) to “really” use it.

Presuming that something is available and in a good working order, a natural question to ask would be: Why would someone chose *not* to use it?

Usually, I do not like figuring out why anyone does anything — it is a fool’s errand, how can one ever know what motivate others; but if I were to speculate in this manner, I’d say that “abandonment” of HiPE could be a consequence of its tight relationship with Erlang language (and, thus, Erlang compiler). 

Approaching the "conclusion” from this direction, and given that HiPE benefits only Erlang (as in Erlang language) community,  it may follow that OTP team’s intention behind this (new) iteration of “JIT” was to provide something that may benefit both, Elixir and Erlang communities.

I must say, I was uneasy when I’ve learned about HiPE abandonment, but cannot say that I can see a downside if experiences acquired with HiPE can be improved upon and presented in a way that benefit both communities.

Of course, this might seem entirely unnecessary if you only use Erlang language (as I do). Not so, if you’re using Elixir to make living. 


Kind regards

V/





> On 20 Jun 2020, at 08:39, Richard O'Keefe <raoknz@REDACTED> wrote:
> 
> From watching the BEAMJIT video, they *tried* building on top of clang, and that turned out to be a mistake.  Then they *tried* building on top of LLVM, and that turned out to be a mistake too.  In fact the detours through Clang and LLVM cost so much time that the whole BEAMJIT idea was very nearly shelved sine die.
> 
> Just because clang & co are getting optimisers &c does not mean that they are getting optimisers that pay off *for Erlang*.
> 
> By the way, I am sick of the way people credit Java with the idea of JITs.
> The idea goes back to load-and-go systems, via Lisp, Brown's invention of "throwaway compiling" for a BASIC system, to Smalltalk (Deutsch & Schiffman), and Self.  Indeed, one issue raised in the video could have been answered by considering Brown's throw-away compiling and the application of that idea in early commercial Smalltalk.
> 
> On Sat, 20 Jun 2020 at 12:35, Felix Gallo <felixgallo@REDACTED <mailto:felixgallo@REDACTED>> wrote:
> Given the highly constrained resources of the OTP team, I would generally expect them to go the other way, and try to hitch their wagons to large systems being built or maintained by others.  The cost of building a de novo high performing JIT and then committing to its ongoing maintenance and development doesn't look, from the outside, like the kind of investment that the OTP team has traditionally made.  Even relatively fundamental subsystems tend to age out for lack of curation.  Is there a new source of funding or staffing which gives the team confidence that they'll be able to make that commitment?
> 
> To ROK's point more specifically -- many new architectures are emerging these days and it's not clear that x86 is the long term winner -- various flavors of ARM, at a minimum, look to be contenders in the near future, at both the consumer and industrial levels.  Clang + co are clearly going to keep getting optimizers and JITs for those environments through the work of thousands of well-incentivized contributors.  What does the BEAMJIT vector look like by comparison?
> 
> F.
> 
> On Fri, Jun 19, 2020 at 3:42 PM Tristan Sloughter <t@REDACTED <mailto:t@REDACTED>> wrote:
> The JIT will support tracing like normal. So even if it was on-par or even a little slower than HiPE I think that is reason enough alone to switch -- and reason enough to have been attempting at alternatives all these years.
> 
> JIT could be a solution that makes sense in general usage while HIPE because of its limitations (even before recent OTP changes that diverged from HIPE even more so) meant it was a niche tool. Very useful to those problems it fit, but still niche.
> 
> Tristan
> 
> On Fri, Jun 19, 2020, at 10:45, Grzegorz Junka wrote:
>> Hi Kenneth,
>> 
>> Thanks to Richard's link I was able to see the presentation about JIT and it looks like the team was putting a lot of effort since as early as 2012 into JIT. But on none of the benchmarks JIT was actually better than HiPE.
>> 
>> I am interested, in what way JIT is better than HiPE that it's worth putting the effort into developing something new instead of fixing HiPE?
>> 
>> Regards
>> 
>> Grzegorz
>> 
>> 
>> 
>> On 18/06/2020 08:35, Kenneth Lundin wrote:
>>> 
>>> HiPE is the runtime and compiler support for native code generation of Erlang modules that some of you might have tried, it is part of the OTP repository today.
>>> 
>>> The OTP team is planning to remove HiPE in the OTP 24 release for the following reasons:
>>> 
>>> we plan to introduce a new way of executing Erlang, the "JIT" described by Lukas Larsson at Code Beam V
>>> since OTP 22, HiPE is not fully functional (does not handle all beam instructions and combinations)
>>> there is no use of HiPE among our primary customers. We actually don't know where HiPE is used except for speeding up Dialyzer which we have another solution for.
>>> The current support for HiPE in the code is a blocker or creates extra work in our new development.
>>> In order to not remove HiPE in OTP 24, we really soon need maintainers committing (long term) to keep HiPE in shape and up to date with the rest of OTP.
>>> 
>>> /Kenneth Erlang/OTP, Ericsson
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20200620/cabcce6f/attachment.htm>