Architectural Suggestions for Job Queuing

[Claes Wikstrom]

Hello,

 > be missing some big picture.  I am looking for are some productive
 > hints.  I want to code the applications myself.
 > 

Ok, I'll just indicate the direction where you should look.

 > The first application I could use would limit the number of
 > simultaneous jobs to run.  In particular, we have limited number of
 > software licenses.  I'd like to start up a server and give it a list
 > of jobs queues and their simultaneous limits.  

This means creating sockets, this is done with gen_tcp.erl

 > Throughout the network
 > whenever someone wants to run job which uses one of those licenses
 > they communicate with the server.  If a license is not available then
 > the job blocks until the license is available.

Typically you would want a concurrent/threaded server here, i.e one
erlang process per opened socket. There are examples of
concurrent tcp/ip servers in the Examples section at
http://www.erlang.org/examples/klacke_examples/index.html

The toplevel process, the one that creates the socket processes could
typically also be responsible for your queues

 > 
 > I already have a program that does this written in ocaml but it uses
 > threads.  Although the threads solution is simple I would prefer to
 > not use them as threads do not allow "introspection".

Erlang process can't be introspected either, you need to send
a message to the process and get a reply, but that is all natural
to the erlang programmer.

>  I would prefer
 > to maintain an explicit queue.  This way the state of the queue could
 > be printed and all of the running and pending jobs could be printed.

This would be done by sending a message to the toplevel 
process and have it report the situation.

The queue is best implemented by the queue.erl module

 > 
 > As an off-shoot of the first application I would then like to write a
 > batch queue system.  Pretty much the only difference is that a general
 > batch queuing system would have a smarter server.  Furthermore that
 > server would invoke the job to run and not just count jobs.  I think
 > if i do a good job of writing the job limiter the batch queue would
 > not be much more complex but just more complicated.

No,

Last but not least, try 
% erl -man queue
or
% erl -man gen_tcp

Which at least on a unix system is a convenient way to view the man pages.


Cheers


/klacke