[erlang-questions] project idea: p2p for large oss mirrors

jm jeffm@REDACTED
Tue Aug 12 04:22:52 CEST 2008

Over the last week I've been attempting to create a large mirror of some 
of the open source projects out there including debian and ubuntu. This 
is quite large well over 200GB. So far it's taken me most of a week to 
do using rsync and it's given me sometime to think about the way this is 
being done. It strikes me as strange that for large mirrors http, ftp, 
and rsync with pull scripts which poll on a regular basis still seem to 
be "best practice" with p2p and other technologies now being available. 
Here's my rough take on a better way to maintain a large mirror.

1. The primary mirror publishes a public key for mirrors on their web site.
2. You down load the primary mirrors public key and configure what your 
interested in mirroring.
3. The primary mirror then publishes update notices detailing the new 
snapshot with a serial number (which could be a date timestamp) along 
with other metadata which is signed. This could be published via rss or 
directly into the p2p system.
4. Your mirror verifies this published notice, and assuming it's valid,
5. Though the p2p system downloads the differences updating the mirror 
in a shadow directory
6. It "flicks the switch" when the mirror reflects the desired state to 
make the update public for ftp/http/etc usage.
7. Optionally publishes an updated status notice (signed of course) so 
that the primary and other mirror sites can maintain a list of mirror 
sites and their status on their own websites.

Thought this might make an interesting project for someone looking for a 
project to do in Erlang but was stuck for ideas on what to do.


More information about the erlang-questions mailing list