[erlang-questions] how to design a scalable 'scheduler' app in erlang way

[Daniel Goertzen]

Your question is a little vague, so I'll just share an Erlang idiom that
I've learned:


- Spawn a new worker process for each request and have it handle the whole
request from beginning to end.  Passing a partially completed request from
scheduler process to worker process is slow[er].

- Use a "server" process to serialize access to the resource.

- The scheduler sounds like a bottleneck.  Remove if you can.  Consider a
worker process that runs like this:

1. Receive request from file, socket or whatever.
2. Spawn a new process, identical to this one, to handle the next request.
3. Process request, using resource as needed.
4. Store/send result.
5. Terminate. (heap gets dropped, no garbage collection needed)


Does that give you some direction?

Cheers,
Dan.

On Thu, Aug 19, 2010 at 2:53 AM, Oscar <ro4tub@REDACTED> wrote:

> hi all,
>
> Here are the scheduler's requirements:
> 1. Scheduler  should receive a lot of requests from other apps.
> 2. Scheduler  should send the received requests to 'worker' apps.
> 3. Scheduler  should determine when to send the received requests to
> 'worker'  apps.
>
> The strategy is:
> a. A request depends on something, named *resource*.
> b. When *resource* is idle, scheduler sends the request to one worker.
> c. When *resource* is busy, scheduler stores the request.
>
> 4. Scheduler should be scalable, supporting about 1million tps.
>
>
> I've implemented it in the shared-state way. TPS is about 100,000, and
> 'lock' consumes the most CPU time.
> Now, I want to implement an erlang version, but have no idea about it.
>
> Shall I store the resource status, pending request to ETS? Will ETS be
> a bottleneck?
>
>
> Thanks,
> -Oscar
>



-- 
Daniel Goertzen
-----------------
dang@REDACTED (work)
daniel.goertzen@REDACTED (home)
-----------------
1 204 272 6149 (home/office)
1 204 470 8360 (mobile)
-----------------