[clc-devel] Making the switch to svn
Hi there, [ As you might have noticed I haven't been around (IRC, ML) last week; this is due to the annual military service I have to attend. It'll continue for another two weeks. However this shouldn't affect any of our plans, Matt, Jay and Per all have the very same permissions on crux.nu as I have; just contact them in case of questions :-) ] To avoid too much duplicate work and further user confusion, I'd suggest to switch the ports system to use the SVN repo from crux.nu soon, and enable the "new contrib" as 'contrib'. Here's a list of things remaining: * Switch to svn: - Write a short wiki page to tell the users what this is all about - Make sure we have all dependencies - pick up ports from danm - contact vkd and Rilo Riemer; ask them to join the svn repo - prepare an update 'ports' port containing core.svup and opt.svup; those can be adapted from http://crux.nu/files/{core,opt}.svn - Make it depend on svup, merge svup from http://xoomer.virgilio.it/srota/ports/sip/svup/ into opt - update 'ports' port in CVSUp; add svup to CVSUp to make it really simple for users to update: - run ports -u - install svup - mv /etc/ports/crux.cvsup /etc/ports/crux_cvsup.old - edit prt-get.conf, replace base with core - ports -u - svn post commit mail for "[security] ..." commits - bug notifications (send to clc-devel for now) * contrib: - provide a proper rsync driver for ports(8); I think Jay prepared one already - reenable the "duplicate view" and statistics in the ports-db on crux.nu Also, we'll need some announcements to the list right before we update 'ports' in CVSUp, and after all is done; especially the "new contrib" could need some more attention :-). After that, we can start with further changes to the website, like checking for missing dependencies on a regular base, implementing a new ports DB (Till ;-)) and adding additional security measures to open "new contrib" to complete strangers (to be discussed). Please append any tasks I forgot to the lists above. Other than that, most of the things remaining are really small things, therefore I'd suggest we plan to make the switch next weekend (december 3rd/4th); however, I as previously mentioned I won't be able to spend too much time on those, so if you can please pick one or more tasks from the list and go for it; please send a quick note to the list to avoid duplicate work. Thank you for your help, Best regards, Johannes -- Johannes Winkelmann mailto:jw@tks6.net Bern, Switzerland http://jw.tks6.net
On 11/26/05 14:52 Johannes Winkelmann wrote:
Hi there,
Hi,
- Make sure we have all dependencies
Quik, dirty, unoptimized one-liner: for f in `prt-get list`; do prt-get depends $f; done|grep from|sort -u (NOTE: put only the new core and opt repos into prt.get.conf)
- pick up ports from danm
I propose maintainers having ports depending on Daniel ones to adopt the deps right now, at least for a short period, in order to facilitate the transition for CRUX users.
- Make it depend on svup, merge svup from http://xoomer.virgilio.it/srota/ports/sip/svup/ into opt
I'll take care of this, I'd like to do some further test and adjustement before committing. On a related note, I'm very happy with svn in general, but not so much with the current svn ports backend, here's the negative aspects (at least compared to cvsup & rsync): 1. Slow / high local load . I'm not talking about network performance, it's just that a 'svn update' seems to hit disk/cpu badly. 2. More clutter into the ports tree, compare: simone@sip: ~ > du -hs /usr/ports/{base,opt,contrib,svn} 2.0M /usr/ports/base 1.9M /usr/ports/opt 17M /usr/ports/contrib 58M /usr/ports/svn 3. quite long compile time for svup (minor annoyance) I've not tested it yet, but zsync[1] could be an interesting backend since it allows rsync synchronization via http. Please share you ideas on the matter. Regards, Simone [1] http://zsync.moria.org.uk/ -- Simone Rota WEB : http://www.varlock.com Bergamo, Italy MAIL: sip@varlock.com
On 11/26/05 16:13 Simone Rota wrote:
I've not tested it yet, but zsync[1] could be an interesting backend since it allows rsync synchronization via http.
My bad, zsync it's intended for single files. -- Simone Rota WEB : http://www.varlock.com Bergamo, Italy MAIL: sip@varlock.com
Hi Simone :) --- Simone Rota <sip@varlock.com> wrote:
On a related note, I'm very happy with svn in general, but not so much with the current svn ports backend, here's the negative aspects (at least compared to cvsup & rsync):
1. Slow / high local load . I'm not talking about network performance, it's just that a 'svn update' seems to hit disk/cpu badly.
2. More clutter into the ports tree, compare: simone@sip: ~ > du -hs /usr/ports/{base,opt,contrib,svn} 2.0M /usr/ports/base 1.9M /usr/ports/opt 17M /usr/ports/contrib 58M /usr/ports/svn
3. quite long compile time for svup (minor annoyance)
Zsync looks interesting. I suppose it's okay that the author still calls is 'beta' - we could certainly explore it. The 'rsync algorithm over http' is pretty much what httpup gives us. So, I'm curious to see if it performs any better. As for rsync, here're the adjustments I made to the existing driver: http://jdolan.dyndns.org/jaydolan/tmp/rsync As you can see, it was simply the addition of the RSYNC_EXCLUDE variable, to prevent source and package archives from being deleted upon `ports -u`. I have not tested it, it was only an idea. As for performing the switch next weekend, I think I'll be around (at least on Saturday) in case the shit hits the fan *g* Jay Dolan Software Engineer, Systems Analyst Windmill Cycles, Inc. 508.999.4000 __________________________________ Yahoo! Mail - PC Magazine Editors' Choice 2005 http://mail.yahoo.com
On 11/26/05 16:35 Jay Dolan wrote:
Zsync looks interesting. I suppose it's okay that the author still calls is 'beta' - we could certainly explore it. The 'rsync algorithm over http' is pretty much what httpup gives us. So, I'm curious to see if it performs any better.
I'm pretty sure it does, even if I I just discovered zsync is for distributing single files, so it's not appropriate here. I'd be more than happy also with a simple solution like making the crux.nu server update the svn repo to a local dir every, say, 5 minutes and serve that dir via rsync (almost the same as with contrib-test). This way we'd have to manage a single (+ more compact and efficient) ports backend. Regards, Simone -- Simone Rota WEB : http://www.varlock.com Bergamo, Italy MAIL: sip@varlock.com
I'd be more than happy also with a simple solution like making the crux.nu server update the svn repo to a local dir every, say, 5 minutes and serve that dir via rsync (almost the same as with contrib-test). Sounds fine with me too; especially if we'll have mirrors of the ports
Hi, On Sat, Nov 26, 2005 at 17:09:58 +0100, Simone Rota wrote: [...] tree in the future, users will have to get used to small delays anyway. Therefore, if the others agree I'd suggest we go for this solution.
This way we'd have to manage a single (+ more compact and efficient) ports backend. Yeah.
I'd be happy though if we could add a wrapper to track synced files, so get rid of the EXCLUDE hack; for now, that should be okay. Kind regards, Johannes -- Johannes Winkelmann mailto:jw@tks6.net Bern, Switzerland http://jw.tks6.net
Johannes Winkelmann wrote:
Hi,
On Sat, Nov 26, 2005 at 17:09:58 +0100, Simone Rota wrote: [...]
I'd be more than happy also with a simple solution like making the crux.nu server update the svn repo to a local dir every, say, 5 minutes and serve that dir via rsync (almost the same as with contrib-test).
Sounds fine with me too; especially if we'll have mirrors of the ports tree in the future, users will have to get used to small delays anyway.
Therefore, if the others agree I'd suggest we go for this solution.
I agree, using rsync is a simple and lightweight solution. I think we could even have instant rsync updates if we updated the svn repo to a local dir using a post-commit hook in subversion.
I'd be happy though if we could add a wrapper to track synced files, so get rid of the EXCLUDE hack; for now, that should be okay.
I wrote an experimental ports driver which does just that (written in perl, but that should be fine since it's included in base). Have a look at: http://www.karsikkopuu.net/crux/misc/rsync. I haven't tested it extensively, but it's working fine so far. It uses a ".checkouts" file in the rsync ports tree to track synced files which must be generated by the repository maintainer every time a change is made, pretty much like httpup-repgen. The command used is: $ find . -depth -not -name . -not -name .checkouts -printf "%P\n" > .checkouts Maybe we could add rsync-repgen to the rsync port is this solution proves worthy of using? Anyway, let me know what you think. // Jukka
Hi, On Sat, Nov 26, 2005 at 17:09:58 +0100, Simone Rota wrote: [...]
I'd be more than happy also with a simple solution like making the crux.nu server update the svn repo to a local dir every, say, 5 minutes and serve that dir via rsync (almost the same as with contrib-test). Just for the record: there's another cool property of this setup: it'll allow us to exchange the VCS if we want to, without the user realizing it, or requiring any additional changes to the files in /etc/ports.
Thanks Simone :-). Johannes -- Johannes Winkelmann mailto:jw@tks6.net Bern, Switzerland http://jw.tks6.net
On Sat, Nov 26, 2005 at 14:52:46 +0100, Johannes Winkelmann wrote:
Hi there, [...] * Switch to svn: - Write a short wiki page to tell the users what this is all about I'll start with that one
- bug notifications (send to clc-devel for now) And look into that one as well
Johannes -- Johannes Winkelmann mailto:jw@tks6.net Bern, Switzerland http://jw.tks6.net
Hi, On Sat, Nov 26, 2005 at 14:52:46 +0100, Johannes Winkelmann wrote:
Hi there, [...] To avoid too much duplicate work and further user confusion, I'd suggest to switch the ports system to use the SVN repo from crux.nu soon
Here's an update: * httpup repositories: - the httpup repos have to be reactivated on crux.nu, for core, opt and contrib (serving the current contrib-test)
* Switch to svn: - Write a short wiki page to tell the users what this is all about Started but not finished yet: http://crux.nu/cgi-bin/trac.cgi/wiki/ServerMigration
- Make sure we have all dependencies - pick up ports from danm Not sure what the state is here
- contact vkd and Rilo Riemer; ask them to join the svn repo Same here
- prepare an update 'ports' port containing core.svup and opt.svup; those can be adapted from http://crux.nu/files/{core,opt}.svn To be done, for rsync though. Should be really easy; I'd suggest to prepare it in 'core' and expose it via httpup; this will allow the users to do $ httpup sync http://../ports/core#ports /tmp/ports $ cd /tmp/ports $ pkgmk -d -i
to update /etc/ports.
- update 'ports' port in CVSUp; Per needs to do that. Haven't heard from him in a while
* contrib: - provide a proper rsync driver for ports(8); I think Jay prepared one already Driver is there, thanks to Jukka.
- reenable the "duplicate view" and statistics in the ports-db on crux.nu To be done
If you happen to have some time please consider doing any of the missing tasks. I should have more time again next week to join you. Thanks, Johannes -- Johannes Winkelmann mailto:jw@tks6.net Bern, Switzerland http://jw.tks6.net
On 12/07/05 22:55 Johannes Winkelmann wrote:
- Make sure we have all dependencies - pick up ports from danm
Not sure what the state is here
Here's the missing deps (one level, there could be missing subdeps as well) faac from ffmpeg gst-plugins from amarok hicolor-icon-theme from scite imlib2 from eterm, ffmpeg, giblib, libast iproute2 from vpnc kdebase from amarok kdelibs from amarok, k3b, kmplayer liba52 from libavifile, transcode libcdio from vcdimager libdevmapper from cryptsetup-luks libdvdread from dvdauthor, transcode libmikmod from sdl_mixer libxslt from docbook-xsl libxvid from ffmpeg, libavifile, transcode psutils from a2ps sdl_image from criticalmass, neverball, wesnoth tcl from ppracer xine-lib from amarok
To be done, for rsync though. Should be really easy; I'd suggest to prepare it in 'core' and expose it via httpup;
I added an updated 'ports' port into attic containing the rsync repos and drivers. I also moved Jue's rsync into core. Didn't touch the one in core yet since I'm afraid some maintainer have a parallel setup with the cvsup ports. Regards, Simone -- Simone Rota WEB : http://www.varlock.com Bergamo, Italy MAIL: sip@varlock.com
On 12/08/05 01:41 Simone Rota wrote: I love self replies :)
I added an updated 'ports' port into attic containing the rsync repos and drivers. I also moved Jue's rsync into core.
Btw, should the cvsup port remain in core? -- Simone Rota WEB : http://www.varlock.com Bergamo, Italy MAIL: sip@varlock.com
--- Simone Rota <sip@varlock.com> wrote:
sdl_image from criticalmass, neverball, wesnoth
I just added sdl_image to opt, as I've been maintaining it in my repository for quite some time anyway. Jay Dolan Software Engineer, Systems Analyst Windmill Cycles, Inc. 508.999.4000 __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com
On Thu, Dec 08, 2005 at 01:41:02AM +0100, Simone Rota wrote:
On 12/07/05 22:55 Johannes Winkelmann wrote:
- Make sure we have all dependencies - pick up ports from danm
Not sure what the state is here
Here's the missing deps (one level, there could be missing subdeps as well)
Simone, thanks for that list.
hicolor-icon-theme from scite
Hmm, that's a problem, because the port vanished into the gnome repo. Matt, could you move it back, please ?
psutils from a2ps
Fixed. Greetings JÜrgen -- Juergen Daubert | mailto:jue@jue.li Korb, Germany | http://jue.li/crux
On Thu, Dec 08, 2005 at 01:41:02AM +0100, Simone Rota wrote:
Here's the missing deps (one level, there could be missing subdeps as well)
faac from ffmpeg gst-plugins from amarok hicolor-icon-theme from scite imlib2 from eterm, ffmpeg, giblib, libast iproute2 from vpnc kdebase from amarok kdelibs from amarok, k3b, kmplayer liba52 from libavifile, transcode libcdio from vcdimager libdevmapper from cryptsetup-luks libdvdread from dvdauthor, transcode libmikmod from sdl_mixer libxslt from docbook-xsl libxvid from ffmpeg, libavifile, transcode psutils from a2ps sdl_image from criticalmass, neverball, wesnoth tcl from ppracer xine-lib from amarok
I will adopt the port libdevmapper. I have checked who is the maintainer of these ports and mostly Nick is the one. I will send him a mail if he needs some help because he seems to be very busy at his study. Regards Viper
participants (6)
-
Jay Dolan -
Johannes Winkelmann -
Juergen Daubert -
Jukka Heino -
Simon Gloßner -
Simone Rota