Hello! I've just tempted to follow http://crux.nu/Main/ServerMigration to get an pretty old CRUX2.1 server updated from the old ports collection to the new one. The ports.prt.tar.gz part works fine but rsync-2.6.7 is not available at samba.org at the mentioned place. It's 2.6.8 now. After updating the Pkgfile and the md5, everything seems to work fine. Can you please update your rsync.prt.tar.gz? Attached are my changes. Beside that, the document is fine! Next step would be a CRUX 2.1 to CRUX 2.2 migration. A link to some documentation for that would be fine... Best greets, -- Clemens Koller _______________________________ R&D Imaging Devices Anagramm GmbH Rupert-Mayer-Str. 45/1 81379 Muenchen Germany http://www.anagramm.de Phone: +49-89-741518-50 Fax: +49-89-741518-19 --- Pkgfile~ 2006-03-13 22:15:05.000000000 +0100 +++ Pkgfile 2006-08-08 10:39:18.000000000 +0200 @@ -4,7 +4,7 @@ # Depends on: openssh name=rsync -version=2.6.7 +version=2.6.8 release=1 source=(http://samba.org/ftp/$name/$name-$version.tar.gz \ rsyncd.conf rsyncd rsync.driver) 082a9dba1f741e6591e5cd748a1233de rsync-2.6.8.tar.gz 7e01fb425c40a8c10c86f3a80aec9e41 rsync.driver a71995f22768c931c5649a1336d25ffb rsyncd b4e95fa8c8f3ae13cfdf616abd6a3960 rsyncd.conf
Hi Clemens, On Tue, Aug 08, 2006 at 11:31:15 +0200, Clemens Koller wrote:
Hello!
I've just tempted to follow http://crux.nu/Main/ServerMigration to get an pretty old CRUX2.1 server updated from the old ports collection to the new one.
The ports.prt.tar.gz part works fine but rsync-2.6.7 is not available at samba.org at the mentioned place. It's 2.6.8 now. After updating the Pkgfile and the md5, everything seems to work fine. Thanks for the heads-up.
Can you please update your rsync.prt.tar.gz? Attached are my changes. Maybe it would be a better idea to mirror the source tarball on crux.nu, since at some point 2.6.8 will disappear as well. I'll do that, and change rsync.prt.tar.gz accordingly.
Beside that, the document is fine! Next step would be a CRUX 2.1 to CRUX 2.2 migration. A link to some documentation for that would be fine... As always, the recommended update procedure is documented in the handbook, available e.g. from http://crux.nu/Main/Handbook2-2 section "Upgrading From CD-ROM". The document is of course also included on the ISOs.
Note that packages linking again older versions of 'db' and 'openssl' will need to be rebuild, use revdep [1] to find the offending ports. HTH, Johannes, who updated his last 2.1 box this morning :-) References: 1. http://jaeger.morpheus.net/linux/crux/files/revdep -- Johannes Winkelmann mailto:jw@smts.ch Zurich, Switzerland http://jw.smts.ch
On Tue, 2006-08-08 at 11:59 +0200, Johannes Winkelmann wrote:
Hi Clemens,
On Tue, Aug 08, 2006 at 11:31:15 +0200, Clemens Koller wrote:
Hello!
I've just tempted to follow http://crux.nu/Main/ServerMigration to get an pretty old CRUX2.1 server updated from the old ports collection to the new one.
The ports.prt.tar.gz part works fine but rsync-2.6.7 is not available at samba.org at the mentioned place. It's 2.6.8 now. After updating the Pkgfile and the md5, everything seems to work fine. Thanks for the heads-up.
Can you please update your rsync.prt.tar.gz? Attached are my changes. Maybe it would be a better idea to mirror the source tarball on crux.nu, since at some point 2.6.8 will disappear as well. I'll do that, and change rsync.prt.tar.gz accordingly.
How about fixing this issue once and for all and mirror all the core/opt distfiles somewhere central and doing wget || in pkgmk, perhaps making the mirror configurable? (Or even better, use a simple user-defined function instead of hardcoding wget + arguments in pkgmk...)
On Tue, 08 Aug 2006 17:19:58 +0200 Mark Rosenstand <mark@borkware.net> wrote:
How about fixing this issue once and for all and mirror all the core/opt distfiles somewhere central and doing wget || in pkgmk, perhaps making the mirror configurable? (Or even better, use a simple user-defined function instead of hardcoding wget + arguments in pkgmk...)
I am sure, you've seen the last message in "[crux-devel] CRUX sources" thread. We can find a few more reasons and re-invent a few more patches (algorithms)... Seems, CRUX's usability suffers sometimes from features unimplemented because of lack of direct interest from main developers. And of coarse, it was expected in such tiny world and I can't blame anybody for that. On the other hand, it could be cowardly called 'simplicity'. Hope, 'Better team organization' will improve the feature requests/patches submission mechanisms. -- Mikhail Kolesnik ICQ: 260259143 IRC: mike_k at freenode/#crux, rusnet/#yalta Jabber: mike_k@jabber.lafox.net NIC handle: MKK83-UANIC
Hi Mike, [ This mail is meant as a general statement, not focused on the mirror patch which seems to have triggered Mike's mail ] On Tue, Aug 08, 2006 at 21:20:09 +0300, Mikhail Kolesnik wrote:
On Tue, 08 Aug 2006 17:19:58 +0200 Mark Rosenstand <mark@borkware.net> wrote:
How about fixing this issue once and for all and mirror all the core/opt distfiles somewhere central and doing wget || in pkgmk, perhaps making the mirror configurable? (Or even better, use a simple user-defined function instead of hardcoding wget + arguments in pkgmk...)
Seems, CRUX's usability suffers sometimes from features unimplemented because of lack of direct interest from main developers. Well, CRUX is developed by this group of people, and in the end, they get to decide what's going in and what's not, because they will have to
[quotation reordered] maintain it. This is how most (if not all?) volunteer projects work. If an idea is well thought out, fixes a general problem and doesn't simply trade in one problem for another, we're certainly willing to discuss it (well, speaking for myself only). However, we only want to change things for the right reasons, not for the sake of change or because $OTHERDISTRO does it that way. Often, omitting features is a design decision, not something we "don't have yet". That said, the presentation of an idea matters a lot, and unfortunately we've been feeling a lot of "us against them" kind of mentality on the mailing lists (yours kinda goes in the same direction, although you probably had only good intentions) and non-constructive complaining, which is rather frustrating for us the (volunteer) developers. At least in my case, this is probably the main reason for the lack of interest in certain people's work; "show no respect, get none in return" holds true in our community just like everywhere else. Counter examples show that things can work well when presented properly: Mark's udev patch which was merged within ~2 weeks, or your networking suggestions which were added to the change list for the next release, and ports for it are already being tested.
I am sure, you've seen the last message in "[crux-devel] CRUX sources" thread. We can find a few more reasons and re-invent a few more patches (algorithms)... However, if a concept has been discussed before, and dismissed as inadequately or better kept as an optional feature, repeating it over and over on crux-devel@ and crux@ without reacting to the concerns usually doesn't help and plainly annoys the developers. You can't talk a feature in, trying to do that usually has the opposite effect.
(This is pretty much what happened in the "CRUX sources" thread you mentioned)
Hope, 'Better team organization' will improve the feature requests/patches submission mechanisms. Independently from the any reorganization, any respected member of the community making modest requests will certainly get a fair discussion about his ideas, especially if he's willing to cooperate; it's been this way in the past, and will be in the future. In exchange, we reserve the right to disagree, and not include a feature if we think it doesn't fit or may as well be optional.
So, that's gotten rather lengthy, however I felt compelled to express this since it's been bothering me for a while already. I hope we can make something positive out of this and, since in the end all we all want is a good linux distribution, right? Regards, Johannes -- Johannes Winkelmann mailto:jw@smts.ch Zurich, Switzerland http://jw.smts.ch
On Wed, Aug 09, 2006 at 01:58:46PM +0200, Johannes Winkelmann wrote: quotations reordered too
[...] since in the end all we all want is a good linux distribution, right?
Sometimes I have the opposite feeling from your side.
Seems, CRUX's usability suffers sometimes from features unimplemented because of lack of direct interest from main developers. Well, CRUX is developed by this group of people, and in the end, they get to decide what's going in and what's not, because they will have to maintain it. This is how most (if not all?) volunteer projects work.
If an idea is well thought out, fixes a general problem and doesn't simply trade in one problem for another, we're certainly willing to discuss it (well, speaking for myself only). However, we only want to change things for the right reasons, not for the sake of change or because $OTHERDISTRO does it that way. Often, omitting features is a design decision, not something we "don't have yet".
Firstly, in case of mirrors, you have to admit: 1. Many crux users have the problem when downloading sources, it is endless discussion on irc. It's endless blaming of sorceforge, gnu.org, and other unstable sites. Many users, especially who are using CRUX in LANs, demand it. It makes life much better. 2. Solutions *exist*. If you are agree on these two clauses, then you have no excuse to not fixing it. Secondly, as for distfiles-like solutions: software must support distfiles-mirrors in first place, and then and only then, there will be own crux mirrors available. If software have no mirrors support, there will never be any mirrors for it. Thirdly, I don't see nothing wrong when using mirrors of other distrubutions, until they are public and don't state nothing against using it in their MOTD.
That said, the presentation of an idea matters a lot, and unfortunately we've been feeling a lot of "us against them" kind of mentality on the mailing lists (yours kinda goes in the same direction, although you probably had only good intentions) and non-constructive complaining, which is rather frustrating for us the (volunteer) developers. At least in my case, this is probably the main reason for the lack of interest in certain people's work; "show no respect, get none in return" holds true in our community just like everywhere else.
Please, don't forget that not only you are the volunteer, we all are. Don't push on that, please. That paragraph is kind of demagogy. If I disagree with somebody, that does not mean that I do not respect him, or I hate him. And conversely, I'm doubt about that if somebody pretending a pussy cat then he are respecting you.
Counter examples show that things can work well when presented properly: Mark's udev patch which was merged within ~2 weeks, or your networking suggestions which were added to the change list for the next release, and ports for it are already being tested.
This is because you had direct interest. I'm still using static dev, thus personally I can't appreciate that.
I am sure, you've seen the last message in "[crux-devel] CRUX sources" thread. We can find a few more reasons and re-invent a few more patches (algorithms)... However, if a concept has been discussed before, and dismissed as inadequately or better kept as an optional feature, repeating it over and over on crux-devel@ and crux@ without reacting to the concerns usually doesn't help and plainly annoys the developers. You can't talk a feature in, trying to do that usually has the opposite effect.
Speaking of me, I have not heard final decision on mirrors patch. All I heard is silent.
(This is pretty much what happened in the "CRUX sources" thread you mentioned)
Hope, 'Better team organization' will improve the feature requests/patches submission mechanisms. Independently from the any reorganization, any respected member of the community making modest requests will certainly get a fair discussion about his ideas, especially if he's willing to cooperate; it's been this way in the past, and will be in the future. In exchange, we reserve the right to disagree, and not include a feature if we think it doesn't fit or may as well be optional.
You are reserve the right to disagree (of course!), but you have not reserved your right to be wrong. That is pity.
Regards, Johannes -- Johannes Winkelmann mailto:jw@smts.ch Zurich, Switzerland http://jw.smts.ch
And please, notice that I'm not talking about your personality: how good or bad person you are. I'm expecting the same or nothing, please. -- Anton (irc: bd2)
On Wed, 9 Aug 2006 18:36:40 +0400 Anton <cbou@mail.ru> wrote: [BLAH BLAH BLAH] There's a saying in open source development circles that comes to mind... "Send patches or shut up" </flamebait> -- Lucas Hazel <lucas@die.net.au> ================================================= "Clothes make the man. Naked men are rarely taken seriously, or given employment." (Mark Twain) =================================================
On Thu, Aug 10, 2006 at 02:29:47AM +1000, Lucas Hazel wrote:
On Wed, 9 Aug 2006 18:36:40 +0400 Anton <cbou@mail.ru> wrote:
[BLAH BLAH BLAH]
There's a saying in open source development circles that comes to mind...
"Send patches or shut up"
http://lists.crux.nu/pipermail/crux/2005-December/005912.html http://lists.crux.nu/pipermail/crux/2005-December/005977.html http://lists.crux.nu/pipermail/crux/2005-August/005261.html http://lists.crux.nu/pipermail/crux/2005-August/005272.html http://lists.crux.nu/pipermail/crux-devel/2006-July/001786.html http://lists.crux.nu/pipermail/crux-devel/2006-July/001807.html http://lists.crux.nu/pipermail/crux-devel/2006-July/001815.html There are also few new patches I've made but not published, because I'm in abyss of hopelessness. I'm going to find good hosting and put there all my work to not bother mailing list with never-will-accepted patches. Good day, -- Anton (irc: bd2)
On Wed, 9 Aug 2006 21:08:30 +0400 Anton <cbou@mail.ru> wrote:
On Thu, Aug 10, 2006 at 02:29:47AM +1000, Lucas Hazel wrote:
On Wed, 9 Aug 2006 18:36:40 +0400 Anton <cbou@mail.ru> wrote:
[BLAH BLAH BLAH]
There's a saying in open source development circles that comes to mind...
"Send patches or shut up"
http://lists.crux.nu/pipermail/crux/2005-December/005912.html http://lists.crux.nu/pipermail/crux/2005-December/005977.html http://lists.crux.nu/pipermail/crux/2005-August/005261.html http://lists.crux.nu/pipermail/crux/2005-August/005272.html http://lists.crux.nu/pipermail/crux-devel/2006-July/001786.html http://lists.crux.nu/pipermail/crux-devel/2006-July/001807.html http://lists.crux.nu/pipermail/crux-devel/2006-July/001815.html
My apologies, I don't actually subscribe to -devel. I suppose what I was saying was misinterpreted (my fault for being too generalised). I suppose what I was really trying to say is if you want something done, do it yourself. Perhaps another quote is in order... "Build them a Jacuzzi and they will come, OK" (That prawn dude from Muppets in Space) Perhaps I'll subscribe to the other lists so I can keep track of these cross-list dicussions :) -- Lucas Hazel <lucas@die.net.au> ================================================= "Clothes make the man. Naked men are rarely taken seriously, or given employment." (Mark Twain) =================================================
On Tue, 2006-08-08 at 11:59 +0200, Johannes Winkelmann wrote: [...]
Maybe it would be a better idea to mirror the source tarball on crux.nu, since at some point 2.6.8 will disappear as well. I'll do that, and change rsync.prt.tar.gz accordingly.
How about fixing this issue once and for all and mirror all the core/opt distfiles somewhere central [...] The main reason this isn't in place is the additional work needed to
Hey Mark, On 8/8/06, Mark Rosenstand <mark@borkware.net> wrote: maintain those distfile mirrors. Combined with the fact that the source availability has been very good recently (my humble opinion as crux _user_) there seemed no real reason to implement a mirroring concept which would add more burden on our shoulders. That said, if there's a volunteer (are multiples ones of course) who'd like to work with us to implement such a system and later maintain it, and could in addition provide bandwidth (amount unknown so far) and HD space for it, we'd definitely do our part to help test it (and in case of a success also make it an official subproject). I'd assume that once setup, the time required for maintenance of this system would be rather low, but during an initial phase, things might be different. If you feel that you'd be in for the above, please contact me :-). Best regards, Johannes -- Johannes Winkelmann jw@smts.ch
Hello, Friends! I see that my mail lead to some interesting discussion... Slighly OT but background thoughts: I think about setting up some remote ports repository + sources + maybe the compiled packages for our very own embedded powerpc hardware to get that somehow "current". So, I just move the Sources to the ports directory right at the time when I test the Port (for my platform) to avoid missing (because of more or less outdated) packages. I guess I'll need to subscribe to crux-devel, too :-) Johannes Winkelmann wrote:
How about fixing this issue once and for all and mirror all the core/opt distfiles somewhere central [...]
The main reason this isn't in place is the additional work needed to maintain those distfile mirrors.
I don't really see much additional work. I think it's sufficient to maintain one master and rsync n slaves.
Combined with the fact that the source availability has been very good recently (my humble opinion as crux _user_) there seemed no real reason to implement a mirroring concept which would add more burden on our shoulders.
The general availability is "okay". I have about 5 machines with CRUX to maintain once in a while. Currently (as of yesterday) windowmaker and gtk don't update... And I guess those are download problems, too - so easily fixable until the urls change again. (I haven't had time for a closer look yet.)
That said, if there's a volunteer (are multiples ones of course) who'd like to work with us to implement such a system and later maintain it, and could in addition provide bandwidth (amount unknown so far) and HD space for it, we'd definitely do our part to help test it (and in case of a success also make it an official subproject). I'd assume that once setup, the time required for maintenance of this system would be rather low, but during an initial phase, things might be different.
I (we) think to get a small root-server later in the project also for our above ppc product if it will get some package management, too. The server project is partly private, partly company related. But I see no problem to mirror a system/structure as long as there is free traffic. But what about an automated ports checker, who polls the dl servers every day once and sees if the port would build successfully? If no -> mail to the mailing list....
If you feel that you'd be in for the above, please contact me :-).
:-) Best greets, Clemens Koller _______________________________ R&D Imaging Devices Anagramm GmbH Rupert-Mayer-Str. 45/1 81379 Muenchen Germany http://www.anagramm.de Phone: +49-89-741518-50 Fax: +49-89-741518-19
Hi, On Tue, Aug 29, 2006 at 15:22:18 +0200, Clemens Koller wrote:
Hello, Friends!
I see that my mail lead to some interesting discussion... [...] Johannes Winkelmann wrote:
How about fixing this issue once and for all and mirror all the core/opt distfiles somewhere central [...]
The main reason this isn't in place is the additional work needed to maintain those distfile mirrors.
I don't really see much additional work. I think it's sufficient to maintain one master and rsync n slaves. You have to trigger updates when a port is updated, and remove unnecessary files after a while (to avoid full HDDs; depending on the quality it's either really easy, or really hard to determine when to remove a file). Once this setup stands, it's probably little work keeping it up, however someone has to initially write and test these things.
I've done some preliminary testing for the triggering, however stopped due to lack of time and need. [...]
I (we) think to get a small root-server later in the project also for our above ppc product if it will get some package management, too. The server project is partly private, partly company related. But I see no problem to mirror a system/structure as long as there is free traffic. Sounds good; what does "as long as there is free traffic" mean here?
But what about an automated ports checker, who polls the dl servers every day once and sees if the port would build successfully? If no -> mail to the mailing list.... Good idea :-) Report from today: http://lists.crux.nu/pipermail/crux-devel/2006-August/001912.html
Summary URL http://crux.nu/files/check_urls.html According to that, the windowmaker problem is to be expected, the gtk isn't there... the tetex server for some reason rejects our requests, but works fine. Thanks for your interest, Johannes -- Johannes Winkelmann mailto:jw@smts.ch Zurich, Switzerland http://jw.smts.ch
Hi, Johannes!
Johannes Winkelmann wrote: You have to trigger updates when a port is updated, and remove unnecessary files after a while (to avoid full HDDs; depending on the quality it's either really easy, or really hard to determine when to remove a file).
I don't necessarily see the necessity to trigger every update. A rsync once a day should be sufficient, when traffic is low. Do you expect problems due to concurrent access to the (master or slave) ports tree? How do other mirrors solve those problems? Pack each port to an archive?
Once this setup stands, it's probably little work keeping it up, however someone has to initially write and test these things. I've done some preliminary testing for the triggering, however stopped due to lack of time and need.
Hmm... But time squeezes all men under it's foot like a bug. Did you publish your approach somewhere?
[...] Sounds good; what does "as long as there is free traffic" mean here?
That only depends on the bucks you pay. -> not decided yet in my case. 10GB HDD and 750GB Traffic for EUR 10.-/Month? on a VServer? If HDD is a problem, for EUR20.-/Month you get a nice dedicated one. Some offers? :-) Well, what's crux.nu's Size at Kalmar NDC AB? What traffic do you have there?
But what about an automated ports checker, who polls the dl servers every day once and sees if the port would build successfully? If no -> mail to the mailing list....
Good idea :-) Report from today: http://lists.crux.nu/pipermail/crux-devel/2006-August/001912.html
Summary URL http://crux.nu/files/check_urls.html
According to that, the windowmaker problem is to be expected, the gtk isn't there... the tetex server for some reason rejects our requests, but works fine.
Here we are... I'm going to subscribe to crux-devel, too. I guess you beat me when I ask you why the problems are not fixed since days? ;-) Greets, Clemens Koller _______________________________ R&D Imaging Devices Anagramm GmbH Rupert-Mayer-Str. 45/1 81379 Muenchen Germany http://www.anagramm.de Phone: +49-89-741518-50 Fax: +49-89-741518-19
Hi again, On Tue, Aug 29, 2006 at 17:19:02 +0200, Clemens Koller wrote:
Hi, Johannes!
Johannes Winkelmann wrote: You have to trigger updates when a port is updated, and remove unnecessary files after a while (to avoid full HDDs; depending on the quality it's either really easy, or really hard to determine when to remove a file).
I don't necessarily see the necessity to trigger every update. A rsync once a day should be sufficient, when traffic is low. Do you expect problems due to concurrent access to the (master or slave) ports tree? Well, I was just thinking about the master. If you run triggered updates, you only have to check those ports which actually changed. If you do it in an interval, you end up checking all ports (well, pkgmk -do for a downloaded port isn't expensive but still). IOW, if there's no change in three days, you have no additional load, while at the same time you get almost instant updates when a change actually happens.
In an interval setup, you have to wait $interval (worst case), and unnecessary load every $interval if nothing changed.
Once this setup stands, it's probably little work keeping it up, however someone has to initially write and test these things. I've done some preliminary testing for the triggering, however stopped due to lack of time and need.
Hmm... But time squeezes all men under it's foot like a bug. Did you publish your approach somewhere? I don't have the files ready right now, but it's pretty simple to describe:
For a particular mail alias, push incoming mail through a script; in this script, extract the svn path. Subscribe to crux-commits with this e-mail address. We can provide a separate mailing list with an easier to parse format if that's needed.
[...] Sounds good; what does "as long as there is free traffic" mean here?
That only depends on the bucks you pay. -> not decided yet in my case. 10GB HDD and 750GB Traffic for EUR 10.-/Month? on a VServer? If HDD is a problem, for EUR20.-/Month you get a nice dedicated one. Some offers? :-)
Well, what's crux.nu's Size at Kalmar NDC AB? What traffic do you have there? Well, we don't host any distfiles there, so those numbers wouldn't have any meaning in this context.
Regards, Johannes -- Johannes Winkelmann mailto:jw@smts.ch Zurich, Switzerland http://jw.smts.ch
Hi, Johannes!
I don't necessarily see the necessity to trigger every update. A rsync once a day should be sufficient, when traffic is low. Do you expect problems due to concurrent access to the (master or slave) ports tree?
Well, I was just thinking about the master. If you run triggered updates, you only have to check those ports which actually changed. If you do it in an interval, you end up checking all ports (well, pkgmk -do for a downloaded port isn't expensive but still). IOW, if there's no change in three days, you have no additional load, while at the same time you get almost instant updates when a change actually happens.
In an interval setup, you have to wait $interval (worst case), and unnecessary load every $interval if nothing changed.
Okay, understood that. I don't have the experience what's happening at crux.nu in detail (would be interesting). But I guess we think about different things now. My idea was to backup the distfiles for the corresponding ports to some server for the case that the distfiles cannot be reached anymore. The pkgmk -d can then fall back to the backup server and grab the distfiles there. That's more or less a proxy behaviour. So, we fill in missing distfiles temporarily in case the real sources are broken. Failed updates due to missing/moving/outdating URLs... are the most inconvenient experience I made with CRUX during the last years. [...]
For a particular mail alias, push incoming mail through a script; in this script, extract the svn path. Subscribe to crux-commits with this e-mail address.
We can provide a separate mailing list with an easier to parse format if that's needed.
Okay, that's a different approach. I was focused on some filesystem-level mirroring + pulled distfiles as a backup server.
Well, what's crux.nu's Size at Kalmar NDC AB? What traffic do you have there?
Well, we don't host any distfiles there, so those numbers wouldn't have any meaning in this context.
Jep, but if you multiply the average number of pulled packages/time times the average used package size we have at least some rough numbers to start with. Greets, Clemens Koller _______________________________ R&D Imaging Devices Anagramm GmbH Rupert-Mayer-Str. 45/1 81379 Muenchen Germany http://www.anagramm.de Phone: +49-89-741518-50 Fax: +49-89-741518-19
Hi, Johannes!
Well, I was just thinking about the master. If you run triggered updates, you only have to check those ports which actually changed. If you do it in an interval, you end up checking all ports (well, pkgmk -do for a downloaded port isn't expensive but still). IOW, if there's no change in three days, you have no additional load, while at the same time you get almost instant updates when a change actually happens.
In an interval setup, you have to wait $interval (worst case), and unnecessary load every $interval if nothing changed.
Okay, understood that. I don't have the experience what's happening at crux.nu in detail (would be interesting). crux.nu hosts the website, subversion for port and tool development and rsync and httpup access to ports. In addition, the mailing lists and bug
Hi, On Wed, Aug 30, 2006 at 13:02:42 +0200, Clemens Koller wrote: tracking are on this machine. No ISO's are hosted there, and it doesn't provide binary packages nor distfiles.
[...]
For a particular mail alias, push incoming mail through a script; in this script, extract the svn path. Subscribe to crux-commits with this e-mail address.
We can provide a separate mailing list with an easier to parse format if that's needed.
Okay, that's a different approach. I was focused on some filesystem-level mirroring + pulled distfiles as a backup server. I think we're talking about different things here. The thread was about establishing a distfile _master_ server. FS level mirroring would be for distfile slaves.
Well, what's crux.nu's Size at Kalmar NDC AB? What traffic do you have there?
Well, we don't host any distfiles there, so those numbers wouldn't have any meaning in this context.
Jep, but if you multiply the average number of pulled packages/time times the average used package size we have at least some rough numbers to start with. Not quite sure what you mean with "pulled packages" here. Remember CRUX is a source based distribution, so packages are build on the end user's machines, not pulled from crux.nu...
Rest assured that if we had a way to calculate the estimated traffic, we would have told you in the very first mail covering this subject. Regards, Johannes -- Johannes Winkelmann mailto:jw@smts.ch Zurich, Switzerland http://jw.smts.ch
Hi, Johannes! I guess it wasn't clear what I've tried to tell/suggest you because I used different nomenclature. One problem I experienced of our non-distfile-hosting structure is that if the distfiles of the remote sources are missing, a package cannot be built. In this case (and only this one) we can let pkgmk -d fall back to a crux-latest-distfile-backupserver (the slave) which had pulled the distfiles from the remote sources once and at the time when the Pkgfiles (on the master(s)) have changed. I would only backup the latest ports trees distfiles and suggest the users to update the ports tree to make sure to use current, available distfiles, too.
Rest assured that if we had a way to calculate the estimated traffic, we would have told you in the very first mail covering this subject.
I don't expect that you tell me the estimated traffic. It would be sufficient to know the current traffic or port download frequency of the port downloads. (rsync, httpup, cvsup) Best Greets, Clemens Koller _______________________________ R&D Imaging Devices Anagramm GmbH Rupert-Mayer-Str. 45/1 81379 Muenchen Germany http://www.anagramm.de Phone: +49-89-741518-50 Fax: +49-89-741518-19
Hi, On Tue, Sep 05, 2006 at 11:07:11 +0200, Clemens Koller wrote:
Hi, Johannes!
[...]
Rest assured that if we had a way to calculate the estimated traffic, we would have told you in the very first mail covering this subject.
I don't expect that you tell me the estimated traffic. It would be sufficient to know the current traffic or port download frequency of the port downloads. (rsync, httpup, cvsup)
Could you maybe just quickly elaborate of the relation between the traffic of the ports tree and the size of the distfiles, or is this number just of general interest to you? Johannes -- Johannes Winkelmann mailto:jw@smts.ch Zurich, Switzerland http://jw.smts.ch
Hi, Johannes! Johannes Winkelmann wrote:
Hi,
On Tue, Sep 05, 2006 at 11:07:11 +0200, Clemens Koller wrote:
Hi, Johannes!
[...]
Rest assured that if we had a way to calculate the estimated traffic, we would have told you in the very first mail covering this subject.
I don't expect that you tell me the estimated traffic. It would be sufficient to know the current traffic or port download frequency of the port downloads. (rsync, httpup, cvsup)
Could you maybe just quickly elaborate of the relation between the traffic of the ports tree and the size of the distfiles, or is this number just of general interest to you?
Yes. It's not about guessing some formula without knowing what data is available, but here you are: The exact bw of such a distfile mirror is computed as: average port updates = bw of the ports tree traffic / average port size Given: ports -u is followed by a prt-get sysup Given: all packages of a port are used / updated (worst case after a fresh install) Given: i.e. 1% of the distfile sources fail average distfile backuprequests = 0.01 * average port updates average distfile mirror bandwith = average distfile backuprequests * average distfile size. q.e.d. ;-) Best greets, Clemens Koller _______________________________ R&D Imaging Devices Anagramm GmbH Rupert-Mayer-Str. 45/1 81379 Muenchen Germany http://www.anagramm.de Phone: +49-89-741518-50 Fax: +49-89-741518-19
Johannes Winkelmann wrote:
You have to trigger updates when a port is updated, and remove unnecessary files after a while (to avoid full HDDs; depending on the quality it's either really easy, or really hard to determine when to remove a file).
I've heard oldfiles in prt-utils works quite well ;)
Once this setup stands, it's probably little work keeping it up, however someone has to initially write and test these things.
I've done some preliminary testing for the triggering, however stopped due to lack of time and need.
Assuming that: - We want to use crux.nu only as a failover mechanism for disappearing sources - We allow secondary mirrors so people can use that as a primary mirror (ie for speed / cost reasons) I think It's not strictly required to have a svn hook; it's very unlikely the source disappears during the short period after the commit and an hypothetical daily cron job that refreshes the sources. Rgarding bandwidth, I don't know how much would rsync consume when secondary mirrors would do their daily job, my guess is not much. Regards, Simone
Hey, On Tue, Aug 29, 2006 at 18:58:31 +0200, Simone Rota wrote:
Johannes Winkelmann wrote:
You have to trigger updates when a port is updated, and remove unnecessary files after a while (to avoid full HDDs; depending on the quality it's either really easy, or really hard to determine when to remove a file).
I've heard oldfiles in prt-utils works quite well ;)
But then you only support those using the very latest ports tree; experience shows that many download problems are in fact slightly outdated ports trees. The older your crux installation/ports tree is, the more you'd appreciate a distfile mirror. Not that we necessarily should encourage keeping an outdated system... That's what I meant with quality. If we provide mirrors for the very latest crux, it's definitely easy. If you want to support 1 or two versions back, some tracking is needed. Regards, Johannes -- Johannes Winkelmann mailto:jw@smts.ch Zurich, Switzerland http://jw.smts.ch
Johannes Winkelmann wrote: Hi,
The older your crux installation/ports tree is, the more you'd appreciate a distfile mirror. Not that we necessarily should encourage keeping an outdated system...
You have a point here.
That's what I meant with quality. If we provide mirrors for the very latest crux, it's definitely easy. If you want to support 1 or two versions back, some tracking is needed.
Yes, that was my idea. Since we're a small team I think it's wiser to concentrate our efforts on keeping the currently 'supported' tree in good shape and do our best to give the nicer possible experience to diligent users. That said, if somebody feels to implement a tool that keeps track of previous versions, I've nothing against using it. Regards, Simone
On Wednesday 09 August 2006 12:12 am, Johannes Winkelmann wrote:
Hey Mark,
On 8/8/06, Mark Rosenstand <mark@borkware.net> wrote:
On Tue, 2006-08-08 at 11:59 +0200, Johannes Winkelmann wrote:
[...]
Maybe it would be a better idea to mirror the source tarball on crux.nu, since at some point 2.6.8 will disappear as well. I'll do that, and change rsync.prt.tar.gz accordingly.
How about fixing this issue once and for all and mirror all the core/opt distfiles somewhere central [...]
The main reason this isn't in place is the additional work needed to maintain those distfile mirrors. Combined with the fact that the source availability has been very good recently (my humble opinion as crux _user_) there seemed no real reason to implement a mirroring concept which would add more burden on our shoulders.
What I do when pkgmk can't find a source is google the filename (for example: curl-7.15.5.tar.bz2), download it manually and build it. I will know if the file is the same because the md5 is compared. Maybe we could do this more automatically: when the source isn't found, use an ftpsearch engine to find the file. Its not 100% foolproof but its better than nothing, and doesn't require any maintainance or additional bandwith usage. Another possibility would be changing pkgmk so that the Pkgfile could include mirrors for its sources. Of course this wouldn't be mandatory, but the packages in core and opt should always have this mirrors, so that they are more reliable. -- Alan
Hello, Alan!
What I do when pkgmk can't find a source is google the filename (for example: curl-7.15.5.tar.bz2), download it manually and build it. I will know if the file is the same because the md5 is compared.
Maybe we could do this more automatically: when the source isn't found, use an ftpsearch engine to find the file. Its not 100% foolproof but its better than nothing, and doesn't require any maintainance or additional bandwith usage.
Another possibility would be changing pkgmk so that the Pkgfile could include mirrors for its sources.
Of course this wouldn't be mandatory, but the packages in core and opt should always have this mirrors, so that they are more reliable.
You got the point. Same story over here. If bandwith would allow it, we could replace the manual google+ftpsearch with an automatic fallback to the crux distfile backup server to make that step more easy. But that's not enough: When the machine is idle, we can build the packages in a sandbox and check for footprint mismatches, too. Best greets, Clemens Koller _______________________________ R&D Imaging Devices Anagramm GmbH Rupert-Mayer-Str. 45/1 81379 Muenchen Germany http://www.anagramm.de Phone: +49-89-741518-50 Fax: +49-89-741518-19
Clemens Koller [2006-09-06 11:05]:
But that's not enough: When the machine is idle, we can build the packages in a sandbox and check for footprint mismatches, too.
CRUX' footprints are quite often failing because of optional dependencies (this being a non-fatal error of course) so I'm not sure whether this particular point is cool ;) Regards, Tilman -- A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing? A: Top-posting. Q: What is the most annoying thing on usenet and in e-mail?
On Wed, 2006-09-06 at 13:45 +0200, Tilman Sauerbeck wrote:
Clemens Koller [2006-09-06 11:05]:
But that's not enough: When the machine is idle, we can build the packages in a sandbox and check for footprint mismatches, too.
CRUX' footprints are quite often failing because of optional dependencies (this being a non-fatal error of course) so I'm not sure whether this particular point is cool ;)
Actually I'm working on a build system that allows for building each package in a completely clean environment (only base + the package's dependencies, the chroot/sandbox is wiped after each build.) This helps to ensure that packages - have the correct dependencies, since the build will fail if they aren't installed at compile time - are less influenced by the host system, so the packages should be faily distributable - are always compiled against the wished libraries, ignore autoconf's "link-against-everything-avaiable" logic Obviously it will take a bit longer to prepare the sandbox for each build, but the base system could be a single tar archive (or some hack like hardlinks + COW or (f)unionfs. Real support for fakeroot is obviously also planned (running the entire compilation in a LD_PRELOAD-hacked environment is a bit insane, only the "make install" and final tar'ing should be fakerooted.)
Mark Rosenstand schrieb:
Actually I'm working on a build system that allows for building each package in a completely clean environment (only base + the package's dependencies, the chroot/sandbox is wiped after each build.)
This helps to ensure that packages - have the correct dependencies, since the build will fail if they aren't installed at compile time - are less influenced by the host system, so the packages should be faily distributable - are always compiled against the wished libraries, ignore autoconf's "link-against-everything-avaiable" logic
Obviously it will take a bit longer to prepare the sandbox for each build, but the base system could be a single tar archive (or some hack like hardlinks + COW or (f)unionfs.
Real support for fakeroot is obviously also planned (running the entire compilation in a LD_PRELOAD-hacked environment is a bit insane, only the "make install" and final tar'ing should be fakerooted.)
when building own ports I found software packages which are doing "evil things" even in the make step, not only in make install. and there are some packages without the classical ./configure | make | make install available - so fakerooting everything should be considered. but i like your overall idea and can provide an even simpler approach, almost without any coding work, using a vserver: - it is a matter of seconds to clone an installed base vserver - fire it up and pkgadd the dependencies for the port in question - optional: for large/common dependencies, clone again - run the actual build on that almost unmodified system - the results can be reportet - overall system modifications during the build are also trivial within the vserver, other conditions can be easily modified: - no network/dns at all (some ports stupidly relay on that) the base vserver could be a regulary bootstrapped installation (based on jaegers crux-2.2-latest.iso for example), or, perhaps even better, a base crux system starting from the last iso release and updated regulary. or both. this way perhaps more hidden build problems which only occur on updated or fresh installs can be detected. keep me informed if you do further steps or like my approach. i am very interested in such a port verification because i mainly use crux as vserver guest os, and this could help improve port quality even for this scenarios. in mid-term, i could also provide vserver-infrastructure. sincerly, cohan (martin koniczek)
participants (10)
-
Alan Mizrahi -
Anton -
Clemens Koller -
Johannes Winkelmann -
Lucas Hazel -
Mark Rosenstand -
Martin Koniczek -
Mikhail Kolesnik -
Simone Rota -
Tilman Sauerbeck