[erlang-questions] newbie: How to go about Implementing this problem

[Joe Armstrong]

Some questions:

- what is the criteria for breaking up the large file? - equal sized
chunks? something else???

- how did the "big" file get onto the machine - big files don't just
appear by magic - it occurs to me
that splitting the file as is created might be better - why bother to
create a big file if all you do is
then split it into smaller chunks. (if the big file was in an nfs file
system you could probably make some
kind of virtual file that did the splitting as is was created, if the
big file came through an FTP transfer
you could write an FTP server that split the file as it arrived, and
so on) - rather than creating a
big file and THEN doing something it would be better to stream the
contents into a splitter and send the fragments to the worker machine
...

You never mentioned it but I assume efficiency and/or fault tolerance
is involved, is this correct?

/Joe Armstrong

On Nov 26, 2007 12:43 AM, Russell King <masterofquestions@REDACTED> wrote:
> Hi guys,
>
> I have looked at gen_server approach of client/server model.  What I
> need to do is to have one application that can create gen_server
> (client/server model) type based on how the initial work chunk is
> divided.  I have to pull data from mainframe.  Basically a big feed
> file created by the mainframe.  I need to break the file in 4 or 6 or
> whatever parts I want and be able to create 4 or 6 or as many servers
> as there are divided parts and dynamically distribute work to the
> servers on different nodes on different machines.  Each server, would
> then create clients on the same machine, to further process their
> assigned part of the file.
>
> for example:
> 1 master(controlling machine1) ,  2 servers (on machine2, machine3)
> get small chunk of the work they are responsible for and each servers
> create clients to process the work.
>
> 1) One, I need to be able to break large work into arbitrary number of
> parts and be able to dynamically create as many servers as there are
> parts (number of parts can be changed on erlang shell and hot code
> loaded). Also need to distribute the server on different machines.
>
> 2) Two, I need to process the mainframe file on repeated basis at
> specified time interval.
>
> Any other solutions to the problem (other than gen_server) are also appreciated.
>
> Russ
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://www.erlang.org/mailman/listinfo/erlang-questions
>