[erlang-questions] A Generic API for controlling software components

[Joe Armstrong]

On Wed, Nov 25, 2009 at 1:40 PM, Jayson Vantuyl <kagato@REDACTED> wrote:
> First, let me take a stab at describing the goal (or what I think is and should be the goal) of components.
>
> It seems that you're trying to address:
>
> 1.  Unifying execution / configuration conventions across popular Erlang projects.
> 2.  Putting some sort of management conventions around deployments of said products.
>
> So, I don't think you're going to get a lot of traction on #2 unless it makes deployment easier.  Right now, those programs all have ad-hoc conventions precisely because Erlang's deployment conventions, while rock solid, are nearly impossible to use outside of an embedded environment.
>
> If you really want to make things better (and I'm sure you do), here's what I suggest doing.
>
> Don't create yet-another-packaging-abstraction (i.e. components).  Applications are perfectly fine for this.  Really.
>

I guess you mean OTP applications? - If so then I disagree. To start
with the directory structure within
an OTP application is fixed which I don't really like. Also there is
no construction rule
that says you are not allowed to write anything in the application
directory tree. I'd like to put the
entire component in a read-only area of disk for safety reasons (once
an initial install has been performed).

Some people want to put components into read-only memory for embedded
devices - it also makes it
easier to scan things for security reasons.

> Address the issues that make it hard to deploy Erlang applications.  Specifically, applications should probably be deployed as a set of modules in a second LIBDIR, that shadows the existing one, thereby reducing the number of duplicates of the entire environment (which is a current problem with Erlang's release stuff).  I already do this, and it's great.

I have a differnt proposal for reducing duplicates - this is a
different problem.

Yes to deploying Erlang components *and upgrading them*

>
> Address issues that make deploying applications hard, in general.  The CLI itch is a good one to scratch.  Make it trivial to write a CLI event handler.  Make it a normal gen_event.  Maybe require a key in the app config (maybe {cli_handler,Module}) or perhaps just have a convention (something like an atom of the form '<application>_cli').  For that matter, make a distribution container that is more like a gem/egg.
>
> I think this is really want you want.  Installing an Erlang application should be as easy as commands something like:
>
> # epkg install rabbitmq
> ...downloads, builds, and installs rabbitmq code somewhere sane...
> # epkg deploy rabbitmq mydeploy
> ...generates a "deployment" somewhere, basically just config and state data, mydeploy is used to generate nodenames (i.e. mydeploy_rabbitmq_main, mydeploy_rabbitmq_helper, etc)...
> # ectl mydeploy start
> ...starts nodes specified for the deploy...
> # ectl mydeploy flush_cache
> ...since flush_cache is not implemented by ectl, it should be passed through to an event handler...

Sure. epkg etc. will be just a thin shell script layer over the API I suggested.
The CLI can be written as a thin layer over the API as well.

>
> Obviously there's a lot to go on under the covers, but anything more complex than that is, at best, a well engineered user-experience failure.  More importantly, I think the above captures what you should be doing.
>
> All of that said, here are my comments on the proposal, so far (most of which I've already stated above):
>
>> What I would like to do is manage all the components in a uniform manner.
>> Much of this design is inspired by how Mac OS-X manages applications.
> Why not just use the application framework, again?  It's 90% there for this.

because 90% is not enough


> Is there anything about an application that makes it unsuitable (i.e. smaller than) a component?  I would just add this to an application.
>
>> By manage I mean start and stop the component, upgrade the code,
>> change the behavior of the component at run time and so on.
> I think you're thinking in terms of actual user-level management, right?  Basically, this is to unify things like ejabberdctl and rabbitmqctl.  I think that's a fantastic idea.

Of course. I just want to download yaws, ejabberd etc. drop them into
my eComponent Directory
and that's it - they are installed and runnable.

>
> Looking forward, I'd suggest taking a hard look at Ruby gems and Python distutils/setuputils/eggs.  I really think that it would make a lot of sense to take the existing Erlang release system and use it to make something not unlike a gem or egg (which it already kind of does, but at a whole-Erlang-system level, not a component level).
>
> As for the command line, it would be extremely handy to be able to have a convention where the nodename of a certain component was known, and there was a command line wrapper to send it commands.  Perhaps, have the component run a gen_event, and have "erlctl <component> <command> [args...]" generate an event in the node that the component runs in.  Extending edoc to generate commandline documentation would be killer here, too.
>
>> Here's a suggestion for a set of rules for managed components:
>>
>>    Draft 1 - 25 Nov 2009
>>
>>    Rule1: All components are unpacked into the same top-level
>>           directory (default $HOME/eComponent) and have the extension
>>           .ec (Erlang component)
>>
>>           Example: Imagine I have installed mochiweb ejabberd and couchDB
>>          then after installation I should see the following:
>>
>>          $pwd
>>           /home/joe/eComponents
>>           $ls
>>           mochiweb.ec ejabberd.ec couchDB.ec
> I don't think eComponents, as a non-hidden top-level directory is going to thrill the aesthetic sense of most Unix users.  I would humbly suggest something more like $HOME/.erlang/components/.  Or, better yet, a system directory (probably just in the Erlang code root) and a user directory ($HOME/.erlang/lib).

Ok, I didn't fully specify this - fine to change the details here or
use an environment variable to point
to the directory.

>
>>    Rule2: A normal user should *never* have to examine any of the
>>           files *inside* the component. To do so causes
>>           "abstraction leakage" - I want to consider each of these
>>           components as black boxes. Once installed the component
>>          should be installed in a read-only disk area.
> Certainly.

This is the bit that is not enforced in OTP applications - I think
users are even encouraged to use the /priv
directory in an otp application

>
>>
>>    Rule3: The component should not break if relocated to a different
>>           top-level directory. A simple "mv" command should suffice
>>          So no hard-wired paths or links please.
> Certainly.
>
>>    Rule4: All components C must have a file called
>>
>>           $HOME/eComponents/C.ec/Preferences.pl
>>
>>          The extension .pl means the file contains a property
>>           list. Here is an example:
> .pl is bad mojo, as it's universally accepts to be Perl.  How about, .epl (i.e. Erlang Property List).

    Ok .eplist then?


>
>>
>>           {codePath, "/bin"}.
>>           {expiryDate, {2009,12,24}}.
>>          {icon, "/images/myIcon.png"}
>>          {version,1}
>>           {myKey1, ...}
>>
>>           The keys codePath, version, and expiryDate are obligatory
>>
>>           Why do we need code path? - so that Erlang can find
>>           a module called C.erl
> If the code path is relative, what is gained by having it specifiable?  Why not just have a convention that it's ebin (or maybe have component/lib be added as an additional LIBDIR).

Because I might not want to follow that particular convention. For
simple applications I usually put everything in one directory.
Actually this could be dropped, the system could scan the filesystem
to find the code paths
the first time you run the program.
directory

>
>>           Expiry date has a "time to live for the component"
>>           Version is used for local configuration data (see later)
>>           Version numbers should start at one and be increased by one
>>           for each new release.
> The expiry time thing is confusing to me.  Is this automatically done by some program, or is it metadata for a person?

It's a hint to tell the package manager how often to check for updates
- I havn't thought through all the details
it could be something else:

     {checkForUpdates, everyTimeYouStart | dayly | hourly | weekly |
{after,Year,Month,Day} | {every, 10, minutes}}

etc.

The designer of the program knows what is a sensible value here

>
>>    Rule5: Local configuration data
>>
>>           Local configuration data must not be stored under the
>>           eComponents directory - (you can't remember we said the
>>           component is in a read-only disk area - see rule2)
>>
>>           Local data for the component C must be stored
>>           in the directory
>>
>>           $HOME/eLibrary/ComponentName/Vsn
>>
>>          Thus local preferences for ejabberd version 1
>>          would be stored in the file
>>
>>          $HOME/eLibrary/ejabberd/1/Preferences.pl
> Again, I humbly suggest more traditional Unix pathnames.  How about $HOME/.erlang/library/<component>/<vsn>/prefs.epl?

No worries ...

>
>>     Rule6: Code upgrade
>>
>>           We should upgrade an component C when it's expiry date has
>>           been reached. To update an component we delete the entire
>>           component under $HOME/eComponents/C.ec we install the
>>           new component and run the command: C:install().
>>
>>           Data that is to be carried over between different versions
>>           of the component is stored in $HOME/eLibrary/C/V/...
> Is this done by the component framework?  Is this really a good idea to be done automatically?  Automatically deployed upgrades have generally been associated with tears and gnashing of teeth in my experience.

I guess the component framework would use the check for updates or
expiry date or whatever
to decide whether to do a check. The check would return a list of
improvements and the user would decide
whether or not to install them.

I guess we should also be able to roll back a version (which is why I
have eLibrary/ComponentName/VSN/data ..
tags.

>
>>     Rule7: management
>>
>>         All components C must provide a module in the file
>>         C_control.erl - so ejabberd provides ejabberd_control.erl
>>
>>         The management API is as follows:
>>
>>         C:start_link() -> Pid
>>
>>            create a controller process for the component.
>>
>>         Pid has the following protocol [see note 1 for notation]
>>         PL is a property list (list of {Key::atom(), Value::any()}
>>
>>          Pid !! {start, PL} => ok | {error, Why}
>>
>>             Cold start the component. PL is a property
>>              list describing how the component should behave
>>
>>         Pid !! {stop, PL} => ok
>>
>>             Stop the component
>>
>>         Pid !! {modify, PL} => ok | {error, Why}
>>             Change the behavior of the component
>>
>>         Pid !! info => PL
>>         Pid !! {info, [Keys]} => [Values]
>>
>>         Pid !! suspendLocal => ok
>>         Pid !! resumeLocal => ok
>>
>>               Suspend the component data can be written to
>>               $HOME/eLibrary/ComponentName/Vsn/...
>>
>>          Pid !! suspendRemote => <<BinaryClosure>>
>>          Pid !! {resumeRemote, <<BinaryClosure>>} => ok
>>
>>                This ton suspend an component on one machine
>>                and resume it on another
>
> This looks a lot like it could just use the existing application framework.  I don't think that most of this would even require breaking backwards-compatibility.

It is pretty similar

/Joe

>
>
> --
> Jayson Vantuyl
> kagato@REDACTED
>
>
>
>
>
>
> ________________________________________________________________
> erlang-questions mailing list. See http://www.erlang.org/faq.html
> erlang-questions (at) erlang.org
>
>