[erlang-questions] project idea: p2p for large oss mirrors
Tue Aug 12 04:22:52 CEST 2008
Over the last week I've been attempting to create a large mirror of some
of the open source projects out there including debian and ubuntu. This
is quite large well over 200GB. So far it's taken me most of a week to
do using rsync and it's given me sometime to think about the way this is
being done. It strikes me as strange that for large mirrors http, ftp,
and rsync with pull scripts which poll on a regular basis still seem to
be "best practice" with p2p and other technologies now being available.
Here's my rough take on a better way to maintain a large mirror.
1. The primary mirror publishes a public key for mirrors on their web site.
2. You down load the primary mirrors public key and configure what your
interested in mirroring.
3. The primary mirror then publishes update notices detailing the new
snapshot with a serial number (which could be a date timestamp) along
with other metadata which is signed. This could be published via rss or
directly into the p2p system.
4. Your mirror verifies this published notice, and assuming it's valid,
5. Though the p2p system downloads the differences updating the mirror
in a shadow directory
6. It "flicks the switch" when the mirror reflects the desired state to
make the update public for ftp/http/etc usage.
7. Optionally publishes an updated status notice (signed of course) so
that the primary and other mirror sites can maintain a list of mirror
sites and their status on their own websites.
Thought this might make an interesting project for someone looking for a
project to do in Erlang but was stuck for ideas on what to do.
More information about the erlang-questions