[erlang-questions] Service platform (longish rant)

[Hal Snyder]

Another observation about pipes.

One of our recent subversive initiatives has been an attempt to  
institute automated testing in a fairly mechanized way using dejagnu  
and expect and such a la gcc and gdb.

The plan is for the toolchain, which already embraces automake, to  
spit out a testframe for each new project (e.g. each new Erlang  
application), making it irresistable for programmers to write/ 
maintain the testsuite as they go.

I naively thought the job would be technically simple as long as you  
have a usable text interface for all the major inputs and outputs.  
Just make sure any GUI-type programmers give you a text alternative,  
etc. Expect may be a bit crufty, but it excels at managing multiple  
dialogues, dealing with Unix process groups and pseudoterminals and  
line disciplines and such.

Wrong. In complex systems, connections among processes under test and  
test drivers have an annoying tendency to plug or stall, and do so  
differently on different runs of the same setup. We have to be a lot  
more careful about issues of flow control and determinism.

I now think of three major components to any data connection where I  
used to think there were only two:
- data format
- protocol / state machine
- flow control

And I have a new respect for Erlang distribution, where  
interconnection is clean and effortless.

Hal

On May 8, 2007, at 5:34 AM, Joe Armstrong wrote:

> I think that all this gets rather complicated because we are thinking
> at the wrong
> abstraction level. I'd really like to see what lies behind the pipes.
>
> When you do "A | B | C" the pipe symbol is really hiding a lot of  
> significant
>  detail. The pipe is really a rather complicated thing which hides  
> all sorts of
> details like flow-control, buffering etc.  I can imagine a more  
> complex notation
>
> Suppose we can "open up" the "|" operator, so we can see inside
>
> The A | B really means
>
>             A |< P >| B
>
> Where P is a process
>
>         P(A, B) := A ? M -> B ! M, P(A,B)
>
> To chain things through pipes I think you need to explicitly state the
> pipe types
>
>       chain([
>                  fun() -> make_a() end,
>                  fun() -> make_standard_pipe() end,
>                  fun() -> make_b() end
>                ])
>
> where make_standard_pipe() makes a simple buffered pipe etc.
>
> What would be really nice would be to implement
> "persistent infinite fault-tolerant with global end-point names" pipes
> (for want of a better
> name) with a defined protocol to get stuff in and out of the pipes
>
> (this would be something like the amazon simple queue service)
>
> Just for fun it might be nice to try and make one of these using a
> defined representation of a pipe
>
> Suppose a pipe descriptor is a tuple like this:
>
> {pipe, Name,  [{Host1,Port2},{Host2,Port2},{Host3,Port2}]}
>
> meaning that pipe called name is stored on three machine accessible  
> through
> {Host, Port}
>
> Then it should be pretty easy to hack together an API to store and  
> retrieve
> tuples to the pipe store (assume also an out-of-band way of  
> distributing
> suitable crypto keys to secure the pipe).
>
> What I havn't thought about is the pipe <-> appliciation protocols
> (some kind or peep/getHead/deleteHead/alertOnNewEntry protocol is  
> needed)
>
> Then you just need a  bit of syntax to expand A | B | ... into
> code that stores and fetches objects from the pipes.
>
> This could make a nice lightweight varient of amazons queue stuff
> without all the
> implementation mess of SOAP etc.
>
> Cheers
>
> /Joe
>
>
> On 5/8/07, Vlad Dumitrescu <vladdu55@REDACTED> wrote:
>> Hi,
>>
>> On 5/8/07, Roger Larsson <roger.larsson@REDACTED> wrote:
>>> On Monday 07 May 2007 22:32, Vlad Dumitrescu wrote:
>>>> The example was
>>>>   </snip>  ...................
>>>
>>> But it can be simplified to
>>> </snip>
>>
>> That was just an example, taken from a previous discussion.
>>
>>> Note that all versions but your process based lacks one component  
>>> from the
>>> Unix pipes - concurrency! The 'ls' is not required to finish  
>>> before the
>>> final 'head', rather when head is finished the pipe is terminated
...