[erlang-questions] Service platform (longish rant)

[Joe Armstrong]

I think that all this gets rather complicated because we are thinking
at the wrong
abstraction level. I'd really like to see what lies behind the pipes.

When you do "A | B | C" the pipe symbol is really hiding a lot of significant
 detail. The pipe is really a rather complicated thing which hides all sorts of
details like flow-control, buffering etc.  I can imagine a more complex notation

Suppose we can "open up" the "|" operator, so we can see inside

The A | B really means

            A |< P >| B

Where P is a process

        P(A, B) := A ? M -> B ! M, P(A,B)

To chain things through pipes I think you need to explicitly state the
pipe types

      chain([
                 fun() -> make_a() end,
                 fun() -> make_standard_pipe() end,
                 fun() -> make_b() end
               ])

where make_standard_pipe() makes a simple buffered pipe etc.

What would be really nice would be to implement
"persistent infinite fault-tolerant with global end-point names" pipes
(for want of a better
name) with a defined protocol to get stuff in and out of the pipes

(this would be something like the amazon simple queue service)

Just for fun it might be nice to try and make one of these using a
defined representation of a pipe

Suppose a pipe descriptor is a tuple like this:

{pipe, Name,  [{Host1,Port2},{Host2,Port2},{Host3,Port2}]}

meaning that pipe called name is stored on three machine accessible through
{Host, Port}

Then it should be pretty easy to hack together an API to store and retrieve
tuples to the pipe store (assume also an out-of-band way of distributing
suitable crypto keys to secure the pipe).

What I havn't thought about is the pipe <-> appliciation protocols
(some kind or peep/getHead/deleteHead/alertOnNewEntry protocol is needed)

Then you just need a  bit of syntax to expand A | B | ... into
code that stores and fetches objects from the pipes.

This could make a nice lightweight varient of amazons queue stuff
without all the
implementation mess of SOAP etc.

Cheers

/Joe





On 5/8/07, Vlad Dumitrescu <vladdu55@REDACTED> wrote:
> Hi,
>
> On 5/8/07, Roger Larsson <roger.larsson@REDACTED> wrote:
> > On Monday 07 May 2007 22:32, Vlad Dumitrescu wrote:
> > > The example was
> > >   </snip>  ...................
> >
> > But it can be simplified to
> > </snip>
>
> That was just an example, taken from a previous discussion.
>
> > Note that all versions but your process based lacks one component from the
> > Unix pipes - concurrency! The 'ls' is not required to finish before the
> > final 'head', rather when head is finished the pipe is terminated.
>
> Yes, that could be a possible requirement. However, there's nothing to
> say that the compiler won't implement all links in the chain (even
> functional ones) as simple processes.
>
> > >     ls() => grep(_£_, "*.erl")   %% _£_ is a reserved variable,
> > >                                          %%     meaning "the current
> > > intermediate value"
> > >           => sort                     %% this is a process name, could be a
> > > pid => hd/1
> >
> > But what is the output from ls()? Complete listing? One line? One char?
> >   ls() => sort => hd/1
> > Same issue for sort, and hd/1... (sort would not like one char at the time...)
>
> That is up to the particular service. Of course, services having
> different format assumptions won't be easy to pipe together. But I
> would personally choose a structured data format over the raw text
> used in the Unix world.
>
> For this particular example, I would have ls() return a list of file
> descriptions if implemented as a function. The result list could be
> lazily constructed behind the scenes, if the framework could handle it
> transparently (by hiding the continuation passing infrastructure).
>
> If ls() is implemented as a process, the result could be returned as a
> sequence of messages containing list chunks, basically forming a lazy
> list too.
>
> [Yes, if ls() returns structured data, the next step shouldn't be
> called 'grep', but again, it is an example]
>
> > > If intermediate values need to be saved, writing
> > >     Term => UnboundVariable => Next
> > > would bind UnboundVariable to the current result value and
> > > UnboundVariable will be accessible in the subsequent scope.
> >
> > This would require the complete result to be passed in each step...
>
> Yes and no. There are two options:
> * Some steps are eager, not lazy. The same applies to something like
> sorting, where it's not useful to sort partial data.
> * Or the value reflects the data sent until the current moment, and
> even sort's output would be the sorted list of partial data. But this
> is an even larger step away from Erlang as we know it, so it would
> take a lot more thinking about what it really means. It feels like a
> different computational model, data-flow based, where the '=>'
> operator is defining a data stream between two entities. I have been
> looking at that too.
>
> I realize that these are delicate matters. I wanted to test the
> response before going further.
>
> best regards,
> Vlad
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://www.erlang.org/mailman/listinfo/erlang-questions
>