Solving the right problem

[Marthin Laubscher]

Spot on. I really should allow the rubber to hit the road now, which I am doing.

 

For the record, what happened in 1991 wasn’t to that I started designing the server I talk about, but that is when I accepted it as my mission to solve a problem I could no longer ignore like others still do. Erlang/OTP was one of many things I filled my toolbox with for use as and when the time comes. 

 

Some of the hurdles and mental blocks I am looking to break through include:-
the apparent manual labour involved in setting up and managing distributed Erlang nodes including setting up physical machines, configuring virtual machines, installing OS, installing Erlang, installing the app, controlling automatic startup and shutdown, plumbing the network and hooking up release management so changes in the experimental code gets rolled out orderly and automatically,
the incredible performance of modern hardware in a single processor node,
the cost and complexity of monitoring virtual machines’ activity on the underlying hardware,
that generating suitable traffic to stress test candidate code might be more troublesome to simulate than it’s worth and
programming isn’t what gives meaning to my life but I love it and tends to get too deeply involved and thus distracted by it from other responsibilities.
 

So by all means, come and join me and my project. Help me overcome and/or sidestep my issues so we can collaborate at the appropriate level on that setting up this adaptive hyper-scaling platform for matching requests and resources to deliver near-real-time responses. We might even call it NoQ, NoMQ or OPR (see, no Q ��, or Optimal Processing Resources).

 

From: Jesper Louis Andersen <jesper.louis.andersen@REDACTED>
Date: Thursday, 7 November, 2019 at 16:48
To: "marthin@REDACTED" <marthin@REDACTED>
Cc: "Erlang (E-mail)" <erlang-questions@REDACTED>
Subject: Re: Solving the right problem

 

On Mon, Nov 4, 2019 at 1:10 PM Marthin Laubscher <marthin@REDACTED> wrote:

Hi everyone,

 

Please pardon my long absences from this awesome (mature) community and my limited knowledge which is likely outdated as well. I’ve known since 1996 when I was first told (in confidence by an Ericsson Radio Systems liaison) about Erlang that it would have to play a role when I eventually get to implementing the system I’ve been working on designing since 1991. That big day is drawing near, so now I’d like to reaffirm my high level understanding of what the language is suited for.

 

If you have been working on this design since 1991, I would recommend you start some experiments into parts of the design. You can often simulate a large amount of devices, and you can also simulate the server side. If there are multiple servers involved, solving the problem in the small scale is often useful: you get to learn about some of the small-scale problems of the design right away. As you scale such a system up, you will run into trouble that has to be solved. But often such solutions stem from the smaller scale ideas you tackled first.

 

At scale, I claim any system breaks down. The real work starts when you reach that level. Almost any web server technology has a breaking point, and you will hit it sooner or later. Erlang has a breaking point and you will hit it sooner or later. It also depends on the nature of the problem you are trying to solve. If you are looking at number crunching, I'd definitely design a hybrid where that is outsourced to some other language suited for fast execution. On the other hand, if you are looking at a problem where you have massive concurrency, you might get away with Erlang or Go. If you add the requirement of robustness on top, at a fine granular level, then I know no other thing than the Erlang stack which can do it. Other languages achieve robustness by containerization of computation and then restarting faulty containers. Yet, the disadvantage of such a method are that errors in such systems can "bleed" to sibling requests. This might not be suitable, depending on the problem at hand.

 

But you really need to start experiments. It is almost impossible to design a software system without the duality of design and implementation helping each other along. That is, design leads to partial implementation. Partial implementation leads to insights. And insights leads to a better design. Over time, it is often the case the design shrinks to a fundamental core. You want that as small as possible as it makes it understandable.

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20191108/8d5375ea/attachment.htm>