[clc-devel] RFC: clc build machine
[Dramatic Prelude] fsck! I'm rewriting this long email after a blackout @ home. My notebook battery suddenly died literally seconds before the power outage and I was left staring at a black screen. The cost of a new battery for my Dell is around 240 euros. Shit. [Back on topic now] I finally managed to dedicate a machine to CLC for port building/testing. We briefly discussed the idea not so long ago in CLC or CRUX Mailing List. Basically it will periodically build and install all ports in /contrib using prt-get for dependencies and log the results. The main aims of this process are essentially two, in order of importance: - Provide a sort of "quality test" for maintained ports: it would be nice to be sure that all ports in /contrib would correctly build and install on a default "blank" CRUX setup. This could be useful for testing proper dependency listing, since it's possible to leave out some deps when building ports on a machine we daily use for "real world" tasks. - Provide binary packages. I asked for this some time ago; while it's a bit premature to add this feature to CLC (and maybe many users/packagers don't like the idea), I will keep binary packages for personal use and eventually upload them somewhere in the net, keeping them separate from CLC. Brief pseudo-code explanation of the process: ------------------------------------------------------ # Initial conditions: only /base ports installed, # with some exception: openssl/h, prt-get, ports, cvsup ports -u activate_giant_fan() activate_liquid_nitrogen_cooling() for port in /contrib: prt-get install 'prt-get quickdep port' # Here we use packages built in this session. # No need to rebuild every package log_single() for port in 'prt-get quickdep port': if port not in /base, /exceptions: prt-get remove port log_all() # html report/summary? move_packages() # move binaries to another loaction so next session # they'll be recompiled ------------------------------------------------------ Questions time: - A confirmation: did we establihsed that /opt ports should be listed as dependencies, didn't we? - This was also briefly discussed before: when a package upgrade (expecially. for libraries) introduces binary incompatibility for dependent packages, we currently don't have a way to make the user aware. In such cases, should we upgrade also the dependent packages (by just increasing release number)? Please note that even if today it's quite easy to "guess" for packages that could need a recompilation, if the port tree will eventually grow, in the future the hunting for such packages could become not-so-trivial. Misc notes: I don't know for sure when the "build machine" will be ready to use, probably sometime next week. I also have no idea how long a full compilation of /contrib ports could take, if the port tree changes "too fast" respect to compilation time many benefits could be lost. (and I could introduce a compile-by-request, compile only changed ports, or similar tricks) If anyone has further suggestions or ideas, it would be gladly accepted. Best regards, Simone
Hi Simone, Just a quick reply for now: On Mon, Sep 29, 2003 at 17:29:45 +0200, Simone Rota wrote: [...]
[Back on topic now]
I finally managed to dedicate a machine to CLC for port building/testing. cool!
The main aims of this process are essentially two, in order of importance:
- Provide a sort of "quality test" for maintained ports: good idea
- Provide binary packages. I asked for this some time ago; while it's a bit premature to add this feature to CLC (and maybe many users/packagers don't like the idea), I will keep binary packages for personal use and eventually upload them somewhere in the net, keeping them separate from CLC. Having binary packages built on a fresh CRUX installation is something I'd probably use to create customized images for my personal use (e.g. no windowaker but blackbox ;-)).
The following is just here for completeness ;-) While I love crux' package management, I fear it might become hairy once binary packages are around; I'm certainly not afraid that you Simone will run into problems (as you know what it's all about), but if Joe User decides to check out crux and wants to install let's say evolution, pkgadd evolution#$VERSION.tar.gz will "just work", while the program won't. I assume there could be scripts to handle this, but it might be hard to forbid all users to use pkgadd directly. The joy of a system without dependencies in built packages ;-) So as a summary I think it's most important to communicate clearly that "binary packages for crux" doesn't mean "RPM-like packages for crux". I'm pretty sure we can reach this goal.
Questions time:
- A confirmation: did we establihsed that /opt ports should be listed as dependencies, didn't we? Yes we did, even though this has probably not been implemented for a large number of ports.
- This was also briefly discussed before: when a package upgrade (expecially. for libraries) introduces binary incompatibility for dependent packages, we currently don't have a way to make the user aware. In such cases, should we upgrade also the dependent packages (by just increasing release number)? Hard to say. I'm not sure what's the right solution WRT the the ideas behind CRUX' package manangement. Release change for binary incompatible is fine with me (as long as we have a good way to trigger this; e.g. the maintainer of a packages with dependent packages must inform the maintainers of the those).
Misc notes:
I don't know for sure when the "build machine" will be ready to use, probably sometime next week.
I also have no idea how long a full compilation of /contrib ports could take, if the port tree changes "too fast" respect to compilation time many benefits could be lost. (and I could introduce a compile-by-request, compile only changed ports, or similar tricks)
If anyone has further suggestions or ideas, it would be gladly accepted. If you did not consider it up to now I'd strongly recommend using ccache. Inbetween minor releases of packages (the software) or new releases of the port, the speed gain is really huge.
Looking forward to this, Best regards, Johannes -- Johannes Winkelmann mailto:jw@tks6.net Biel, Switzerland http://jw.tks6.net
On Mon, 29 Sep 2003 18:14:09 +0200 Johannes Winkelmann <jw@tks6.net> wrote:
Hi Simone,
Hi Johannes,
While I love crux' package management, I fear it might become hairy once binary packages are around; I'm certainly not afraid that you Simone will run into problems (as you know what it's all about), but if Joe User decides to check out crux and wants to install let's say evolution, pkgadd evolution#$VERSION.tar.gz will "just work", while the program won't. I assume there could be scripts to handle this, but it might be hard to forbid all users to use pkgadd directly. The joy of a system without dependencies in built packages ;-) So as a summary I think it's most important to communicate clearly that "binary packages for crux" doesn't mean "RPM-like packages for crux". I'm pretty sure we can reach this goal.
I can see your point and agree that the users shoul be clearly informed that pkgadd wouldn't suffice for CLC binaries. Incidentally, this is still valid for /opt packages, for example. I thought of a prt-get switch or something similar: prt-get install `prt-get quickdep kdebase` --binary the --binary option would download packages into their dirs (or pkg_dir), then proceed as usual (the binaries are up-to-date and recompilation is not needed). PS: what about a "prt-get depinst portname" command that would automate the quickdep stuff?
- A confirmation: did we establihsed that /opt ports should be listed as dependencies, didn't we? Yes we did, even though this has probably not been implemented for a large number of ports.
So i must be prepared to see a long list of failed installations in the first log ;-)
- This was also briefly discussed before: when a package upgrade (expecially. for libraries) introduces binary incompatibility for dependent packages, we currently don't have a way to make the user aware. In such cases, should we upgrade also the dependent packages (by just increasing release number)? Hard to say. I'm not sure what's the right solution WRT the the ideas behind CRUX' package manangement. Release change for binary incompatible is fine with me (as long as we have a good way to trigger this; e.g. the maintainer of a packages with dependent packages must inform the maintainers of the those).
Good, even if I'm not sure about the maintainer's responsability (he should have a clue about dependent packages, this could require some work; anyway prt-get dependent should help a lot). It'll be fine to hear from other maintainers too about the 'fake' upgrade issue. Problems arise when new libs are not declared as binary incompatible by their developers while in fact they are: I had some issue in the past with the fox toolkit: a minor upgrade to the lib broke some app. But it could just be that I'm extremely unlucky.
If you did not consider it up to now I'd strongly recommend using ccache. Inbetween minor releases of packages (the software) or new releases of the port, the speed gain is really huge.
Unfortunately I had some issue with ccache in the past: some port (icewm, ruby if I remember well) failed to compile; disabling ccache solved the prblem.
Looking forward to this, Best regards, Johannes
Thank you very much for your reply, Regards, Simone
On Mon, 29 Sep 2003 18:14:09 +0200 V> Johannes Winkelmann <jw@tks6.net> wrote:
Hi Simone,
Hi Johannes,
While I love crux' package management, I fear it might become hairy once binary packages are around; I'm certainly not afraid that you Simone will run into problems (as you know what it's all about), but if Joe User decides to check out crux and wants to install let's say evolution, pkgadd evolution#$VERSION.tar.gz will "just work", while the program won't. I assume there could be scripts to handle this, but it might be hard to forbid all users to use pkgadd directly. The joy of a system without dependencies in built packages ;-) So as a summary I think it's most important to communicate clearly that "binary packages for crux" doesn't mean "RPM-like packages for crux". I'm pretty sure we can reach this goal.
I can see your point and agree that the users shoul be clearly informed that pkgadd wouldn't suffice for CLC binaries. Incidentally, this is still valid for /opt packages, for example. I thought of a prt-get switch or something similar:
prt-get install `prt-get quickdep kdebase` --binary
PS: what about a "prt-get depinst portname" command that would automate the quickdep stuff? It used to be there and I removed it as I was afraid that it could imply
Hi, On Mon, Sep 29, 2003 at 18:57:25 +0200, Simone Rota wrote: this will not prevent Joe from doing prt-get install evolution --binary which will work fine... so prt-get should in this case print a warning "dependencies not satisfied". This is a dependency _checking_, something I don't want to do as long as we have informal dependencies. I was thinking of a new tool (another tool for a different kind of job) codenamed bininst which downloads and installs the package, using either a dependency matrix as http://sto.f2o.org/crux/wiki/DependencesMatrix when there's no ports tree around (think: installation), or prt-get's quickdep. Something like: bininst [-d deps.matrix] -u http://clc.berlios.de/packages evolution bininst would do the following: for each package from the dependency list: download from server (-u; could be configured in bininst.conf) pkgadd || exit on error Of course this could be integrated in prt-get, but for now I'd like to get a feeling of this first and I think a dedicated tool might help to distinuish as binary install vs. ports tree install are quite different. I hope I'll find some time this week to write a small prototype. that dependencies were working. You know, people usually don't bother reading the man page or even the manual, and it would be kind of frustrating to get "bug reports" about prt-get failing to install a package. The current way moved the responsibility away from me to the Packager ;-). Anyway, I guess I'll reintroduce it in one of the future versions. Regards, Johannes [...]
If you did not consider it up to now I'd strongly recommend using ccache. Inbetween minor releases of packages (the software) or new releases of the port, the speed gain is really huge.
Unfortunately I had some issue with ccache in the past: some port (icewm, ruby if I remember well) failed to compile; disabling ccache solved the prblem. Same here, OTOH those are only a handful of packages, and when I build new packages (as user first, rebuild with removed docs, root built) then ccache saves me a lot of time.
Regards, Johannes -- Johannes Winkelmann mailto:jw@tks6.net Biel, Switzerland http://jw.tks6.net
On Tue, 30 Sep 2003 16:22:47 +0200 Johannes Winkelmann <jw@tks6.net> wrote:
I thought of a prt-get switch or something similar:
prt-get install `prt-get quickdep kdebase` --binary this will not prevent Joe from doing prt-get install evolution --binary which will work fine... so prt-get should in this case print a warning "dependencies not satisfied". This is a dependency _checking_, something I don't want to do as long as we have informal dependencies.
I support your decision not to check for dependencies if the user didn't explicitly asked for the feature from the commandline. Anyway I think we're discussing a non-issue: a CRUX user *should* be aware that installing a package without its deps is something that doesn't work. It is true, as you pointed out, that a source install would immediately reveal the missing dep while a binary install would not. Personally it's something I can live with, since it's easy with prt-get and our 'unofficial' dependencies listings to handle potential problems.
I was thinking of a new tool (another tool for a different kind of job) codenamed bininst which downloads and installs the package, using either a dependency matrix [cut]
Of course this could be integrated in prt-get, but for now I'd like to get a feeling of this first and I think a dedicated tool might help to distinuish as binary install vs. ports tree install are quite different. I hope I'll find some time this week to write a small prototype.
That would be a great tool. Having it developed as a standalone util has its advantages; more than this, you could eventually wrap the tool from prt-get (as you do with pkgmk) in the future. I'd prefer to see the binary install feature integrated with prt-get and use prt-get as the all-in-one tool for package management on CRUX, but this is a matter of personal taste. There are pros/cons in both solutions, one of this is that the port tree is needed for binary installs if the feature is to be integrated in prt-get. On the other side, we'd have to maintain an extra dependencies chart and the user would at least need an 'external' file. Reasoning strictly from the user's point of view, I think a simple --binary switch in prt-get would be a simpler and "more understandable" solution. Ok, I'll stop my rantings about personal preferences, whatever solution you prefer it would be a great addition to CRUX (ps: thank you very much for your past, present and future work!)
PS: what about a "prt-get depinst portname" command that would automate the quickdep stuff? It used to be there and I removed it as I was afraid that it could imply that dependencies were working.
What?!? Don't they? :-)
You know, people usually don't bother reading the man page or even the manual,
I for one, since I missed the depinst command in previous releases (or maybe I inconsciously asked for a feature I saw at the time and was recently removed)
Anyway, I guess I'll reintroduce it in one of the future versions.
great, I'll use a bash wrapper for the moment. [ccache]
Same here, OTOH those are only a handful of packages, and when I build new packages (as user first, rebuild with removed docs, root built) then ccache saves me a lot of time.
True, not to count the dozens of Pkgfile adjustements I do before a release ;-). ccache is really a time-saver.
Regards, Johannes
Best regards, Simone
On Mon, Sep 29, 2003 at 05:29:45PM +0200, Simone Rota wrote: [...]
- Provide binary packages. I asked for this some time ago; while it's a bit premature to add this feature to CLC (and maybe many users/packagers don't like the idea), I will keep binary packages for personal use and eventually upload them somewhere in the net, keeping them separate from CLC.
Why do we need binaries ? To spare the user from compiling the packages from source, to have a "complete" system up and running as fast as possible whithout any knowledge ? Are we going away from "experienced user" to mainstream distri ? Just my two cents ;) regards Jürgen -- juergen.daubert@t-online.de
On Tue, 30 Sep 2003 18:09:30 +0200 Juergen.Daubert@t-online.de (Juergen Daubert) wrote:
On Mon, Sep 29, 2003 at 05:29:45PM +0200, Simone Rota wrote: [...]
- Provide binary packages. I asked for this some time ago; while it's a bit premature to add this feature to CLC (and maybe many users/packagers don't like the idea), I will keep binary packages for personal use and eventually upload them somewhere in the net, keeping them separate from CLC.
Why do we need binaries ? To spare the user from compiling the packages from source, to have a "complete" system up and running as fast as possible whithout any knowledge ? Are we going away from "experienced user" to mainstream distri ?
Hi Jürgen, thanks for your reply. IMHO binary packages could be useful under certain circumstances: - First, They save a lot of time. For experienced users too ;-) - Machines without gcc: think of thin clients or production environments (or always-online machines: firewalls, servers...) where it's preferred not to have a build environment available for security reasons. - Quick setup of testing machines. - And so on... I understand that you could still not see any benefit of using binaries in the scenarios described above. What I don't completely understand is the binary<->no knowledge association. I don't think a newbie could actually learn something just by watching the output of pkgmk. CRUX has been for me a great way to learn something about Linux, but this is totally unrelated to the fact that it is a source-based distro. The fact that manual configuration is needed for everything you install is the key feature, and this has nothing to do with the way I installed the package.
Just my two cents ;)
It's always good to hear different opinions; regarding binary packages I anticipated there would be "negative" reactions, that's why the project will be developed outside CLC.
regards Jürgen
Best regards, Simone
participants (3)
-
Johannes Winkelmann -
Juergen.Daubert＠t-online.de -
Simone Rota