Architectural Suggestions for Job Queuing
John Gerard Malecki
Wed Apr 14 17:21:35 CEST 1999
Thanks for the response. It looks like there does not exist a
gen_batch_queue or gen_software_license module which would immediately
solve my problem. :)
Getting on to more specifications of the applications let me first
state the following. I have no intention of selling the programs or
running them outside of the company where i work. If they turn out to
be generically useful I'd gratefully make them available to the public
I'm not sure how much work is involved in developing a robust (crash
resistant) and reasonably efficient application. I am curious about
> You have to decide:
> 1) Server in Erlang + clients in Erlang + Distributed Erlang for
> communication, or,
> 2) Server in Erlang + clients in Erlang + socket communication,
> 3) Server in Erlang + clients in (C, C++, ...) + socket communication
> + API for (C, C++, ..) client side applications
> These are in order of complexity (simplest first).
> 1) would be OK for a quick prototype to get the protocols ok
> 3) would be for a commercial product that you could sell
> 2) is a half way house to implementing 3)
I figured that implementation #1 would minimize the amount of my work.
Why is option #2 "better" than #1? I presume you are saying that #3
is good for a commercial product as the end-user could interface to it
using C and not Erlang. When you mention option #2 are you suggesting
that there is some good reason for using socket communication in
preference to Distributed Erlang communication?
> I think the key architectural/design problem is one of deciding
> what you want to do in the event of failure. If the server crashes,
> then all the clients are blocked. If there is a communication failure
> then you may loose licenses etc.
I agree. This may sound selfish but I would like to implement a
robust distributed application because it sounds like fun and i have
never written one before.
> Here you can virtually design whatever you want.
Darn, this is where i was hoping for a gen_software_license module :)
> In a small (closed) system - then maybe there should be no server -
> the clients could alternatively take on the roles of client or server and
> use a broadcast/lock strategy to negotiate licenses.
> In a commercial system all nodes might not be equal. One server
> (placed on a reliable node) might service hundreds of clients, etc. -
> you still might want to have some hot-standby/fail-over behaviour for
> the server,...
> This kind of stuff soon gets complicated (but that's OK - our
> telecomms stuff is like this :-)
Since this is not a commercial system i have a lot of flexibility
here. I am not familiar with the trade-offs. I presume that the more
robust the system the more work it will take to get it written.
> Firstly I'd, like to see a simple "ball park" analysis of the
> problem, in terms of;
> a) How many clients
We have around 100 workstations at our company. This sounds like a
> b) How many servers
Other than for reliability or performance i see no reason why the
answer could not be 1. Furthermore, I see no reason why the server
could not run on all of the machines -- all 100 of them.
> c) Holding times (how long does a client use a license)
Rarely less than 10 seconds. Normally a minute or two. There may
also be interactive use where the license could be held for hours at a
time. There may also be large batch jobs where the license is held
for some hours.
> d) Reliability requirements.
> - Is it acceptable that clients block if the server crashes?
> - Do you want hot standby for server crashes?
Yes and yes. I'm not familiar with the coding and performance impact
of having a hot standby. I have never implemented anything like this
> e) is this a LAN/WAN application?
> f) required response times for obtaining/freeing a license
Less than 1 second.
> g) security levels (none, ... full) How much effort do you
> want to put into making sure that you cannot forge a license
> (you can have anything up to a full public/private key system)
None. Environment is completely secure. (Ha!)
> h) Maintaince levels (do you want a remote management system)
> If so what ...
I'm not sure what remote means but yes i do want the ability to
dynamically increase or decrease license counts, add new queues, shut
down queues, etc. This is one of the reasons Erlang intrigued me as
it seems that you guys have familiarity with these kinds of
> Once you have some idea of the answers to questions like these you
> can *begin* to think about an architecture.
> It may be that you have a very specific set of answers - fine then
> we can talk architectures. Or, you might want to "grow" a solution for
> a very simple idealized system.
I'm in no rush. I like the idea of having an architecture in mind,
coding a pilot version, getting feel for how well it will work, making
changes and then when happy adding all the bells and whistles.
By the way, here is a more detailed description of how my license
client and server work. This works well enough in the environment
which i have described above.
My present implementation of the license client in ocaml is quite
small. When the user wants to execute a command on a named license
queue, the user enters
license_client name command
The license_client program communicates with the server by sending it
the license name. The license_client program then does a read and
blocks. When the server decides that the named license is available
to run it writes to the client. The pending license_client read now
succeeds and then the license_client executes the desired user
Once the user command finishes the license_client closes the
connection with the server. The server now knows that the license is
available. This approach works well as licenses are rarely lost even
if the client is killed or crashes. If the server goes down i force
the clients to wait. If the server goes down in the middle of a
transaction then there is a problem. This is an area for improvement.
Since the present implementation of the server uses threads it is
convenient to only remember the currently running processes. It
messes up the code quite a bit to create a queue of pending processes.
This is however a desirable feature and so i would like to add this in
the next implementation.
More information about the erlang-questions