[clc-devel] Rsync for Maintainers

Johannes Winkelmann jw at tks6.net
Wed Sep 1 19:39:23 UTC 2004


On Wed, Sep 01, 2004 at 12:07:34 -0400, Victor wrote:
> Johannes Winkelmann wrote:
> >Hi,
> > 
> [snip]
> >
> >Advantages:
> >- Changes are transmitted using diffs (httpup: whole files)
> Well, also clients don't have to run any servers at all.
I mentioned that in the difference section; this is both advantage and
disadvantage; if you want an httpup repo, this solution would duplicate
your work; furthermore and you can't develop independently. I'm not
saying that distributed development is better, but I'm not convinced
that centralized development is.

Note that if you join CLC, you don't need a repo :-) 

> There is a central location for all the ports (fewer downtime with 
> people rebooting their boxes, not everyone has dedicated servers)
Okay, I guess I wasn't too clear. Both suggestions I made would be
hosted on a central server as well. Central access, but distributed (=
independent) development.

Illustration of the 'people collection' idea:

                     | central server
+--------------+      sync      +----------------------+
+ httpup repos + -------------> + separate collections +
+--------------+  n          1  +----------------------+
    ^                |                     |
      possible to subscribe directly       | - merge into one collection
    | to prefer a repo over base/opt/      | - check duplicates
      contrib, or if you don't want        | - rm base/opt/contrib dups
    | everything from people               | - whatever checks you want
                     |                     | - notifications
    |                |                     V
+------+     fetches         +----------------------------+
+ User +  <--------------    + httpup collection 'people' + <-> Mirrors
+------+  n              1   +----------------------------+

Note that this even allows maintainers of private repos to have
duplicates over base/opt/contrib (my former mail said that such
duplicates would cause a notification, coupled with the request to remove
it); they just won't be propagated to the 'people' collection. Users can
still get those by subscribing to the repo directly, and putting it
higher in prt-get.conf than /usr/ports/people. At the same time, people
who have dups over ports from base/opt/contrib don't have to maintain
two httpup repos.

I guess there is a need for a better explanation; I'll try to put it 
together sometime soon.

> >- If a repo maintainer decides to just not update his repo anymore, we
> >  have to detect that he's not accessing our repo anymore; this is a bit
> >  harder in a "push" model than it is in a "pull" model (as the httpup
> >  ideas implement).
> Yeah, that's an issue... How is it solved now?
the sync script updates the collections on the central server hourly (or
whatever interval we'll choose). If there's a 404, we'll find out
instantly. If we want to introduce checking for changes, this can be
implemented in this update script.

> >- To merge those ports into one collection, you'll need another script
> >  (I don't think rsync does this); therefore, you'll end up with more
> >  components to maintain (rsync, script), which run independently, which
> >  means that you have to manually ensure that there are no "commits" to
> >  the collection while merging it (I know this is simple to do, but it
> >  will cause some disk load).
> Actually, I didn't think we would merge them into one collection.
Okay, but you need to check for duplicates, right? Unless you do that,
it'll always be a number of independent repos, thus the quality won't
improve which is one of my main goals.

Regards, Johannes
Johannes Winkelmann              mailto:jw at tks6.net
Bern, Switzerland                http://jw.tks6.net

More information about the crux-devel mailing list