[clc-devel] Future development of CRUX / pkgutils discussions
Hi there, The dust after the 2.2 release has settled a bit, and I'd say it worked out pretty well. Thanks for your efforts in making that a (relatively) painless update. I've started two wiki pages to discuss the future development of CRUX itself and pkgutils, meant to serve as a outline for IRC meetings. Those pages are: http://crux.nu/Main/DevVision http://crux.nu/Main/PkgutilsIdeas I'd appreciate if anyone with write access would add his notes to those pages. Regarding DevVision: This page should serve to define a common vision and future goals of CRUX. Obviously, this is subject to frequent change, but I'd like to nail down some basic tasks and goals to make sure we're on the same page, and we're not wasting time implementing things the majority of devs doesn't want. In addition, this should allow people interested in certain subtasks to start working on it early on, seeing that we i.e. want something for the 2.3 release. Regarding PkgutilsIdeas: It's a rather longish document; if you're not that much interested, please jump directly to the "Summary" section. There's a section for personal comments and goals, and I'd be happy if those of you who have expressed an interest in pkgutils hacking (that would be at least sip, jdolan, tilman and vektori) to add their ideas there as well. Also related to this, I wrote a prototype (read: code not meant to be used in a final product but to demonstrate a concept) for attribute based pkgutils, including meta data in binary packages. There's an extra page for this at http://crux.nu/Main/PkgutilsAttributes, including a section on how to get the code and play with it. I'd suggest to have two separate IRC meetings (pkgutils first, the "vision" second) before the end of May; dates are subject to discussion. Comments both on the wiki pages and the rest of this mail are highly welcome. Kind regards, Johannes -- Johannes Winkelmann mailto:jw@smts.ch Zurich, Switzerland http://jw.smts.ch
On 04/25/06 11:15 Johannes Winkelmann wrote:
I've started two wiki pages to discuss the future development of CRUX itself and pkgutils, meant to serve as a outline for IRC meetings. Those pages are: http://crux.nu/Main/DevVision http://crux.nu/Main/PkgutilsIdeas
I'd appreciate if anyone with write access would add his notes to those pages.
I'd suggest to have two separate IRC meetings (pkgutils first, the "vision" second) before the end of May; dates are subject to discussion.
Comments both on the wiki pages and the rest of this mail are highly welcome.
Hi, here's a couple of first impressions, esp. on the new pkgutils, before I collect enough info / ideas to edit the wiki page: - Independently from the features we'd like to push, I'm in favour of a rewrite; I suggest either plain C with libtar or sh, with portability in mind. I'm in the middle of a ANSI C pkgutils port, (pkginfo, pkgrm almost done), my first impressions while working on the code is that using sh would significantly speed up developement and make it easier to add features in the future. - I'm a bit worried about the metadata proposal: - Not sure if leaving free, per-repository attributes can make things easier rather than produce confusion / lack of standards - Having metadata in the package means that when such fields has to be corrected (ie: typo in dependencies), the package has to be rebuilt and redistributed. Regards, Simone -- Simone Rota Bergamo, Italy - http://www.varlock.com
On Tue, 2006-04-25 at 13:49 +0200, Simone Rota wrote:
I'm in the middle of a ANSI C pkgutils port, (pkginfo, pkgrm almost done),
Nice :) It's my impression that different people are working individually on private copies of a pkgutils port in C. Perhaps setting up a temporary repository could unite the forces?
my first impressions while working on the code is that using sh would significantly speed up developement and make it easier to add features in the future.
Except for the initial development time, it sounds like C is the way to go since it's pretty good at preventing featurism (which is much better in the "upper layers", e.g. prt-get) Personally I'd (hopefully) find a C implementation more reliable, but maybe that's just be me.
- I'm a bit worried about the metadata proposal: - Not sure if leaving free, per-repository attributes can make things easier rather than produce confusion / lack of standards
Agreed.
- Having metadata in the package means that when such fields has to be corrected (ie: typo in dependencies), the package has to be rebuilt and redistributed.
While I get your point and share your worries, using ar archives with metadata + tar.gz for the actual data would make this a non-issue. Or very close at least :)
Mark Rosenstand wrote:
It's my impression that different people are working individually on private copies of a pkgutils port in C. Perhaps setting up a temporary repository could unite the forces?
But then how could one person hijack the project and win recognition? An attempt to form a baseline C design which was to be openly discussed from day one, rather than publicly previewed once in beta stages of implementation, was already put forth and immediately rejected. -- Jay Dolan jdolan.dyndns.org A: Because it's annoying to read. Q: Why is top-posting bad? A: Top posting. Q: What's the most annoying thing about usenet?
Hi Jay, On Tue, Apr 25, 2006 at 19:56:15 -0400, Jay Dolan wrote:
Mark Rosenstand wrote:
It's my impression that different people are working individually on private copies of a pkgutils port in C. Perhaps setting up a temporary repository could unite the forces?
But then how could one person hijack the project and win recognition? o_O
An attempt to form a baseline C design which was to be openly discussed from day one, rather than publicly previewed once in beta stages of implementation, was already put forth and immediately rejected. What exactly are you talking about? Simone hasn't published any code so far (at least not to my knowledge). The attribute stuff is explicitely marked as not being meant to be used in the final implentation but to show how attributes could work (sidenote: attributes is a concept we've been talking about since CRUXCon 2004).
The only person who has so far posted code and suggested it to be used for future pkgutils was you (Jay) [1], and it hardly qualifies as an implementation in beta stages since it was just a c header file. When I wanted to discuss it and dared to question your approach to use a C header as starting point for a _design_ discussion, you got offended ("..forget i posted it. you do it.") and figured it would be a good idea to attack my coding efforts ("but i've seen enough of your code to say with confidence that you do not always execute planning strategy"). Now this... what are you trying to achieve? Confused, Johannes References 1. http://jdolan.dyndns.org/jaydolan/tmp/libpkg.h -- Johannes Winkelmann mailto:jw@smts.ch Zurich, Switzerland http://jw.smts.ch
Johannes Winkelmann wrote:
An attempt to form a baseline C design which was to be openly discussed from day one, rather than publicly previewed once in beta stages of implementation, was already put forth and immediately rejected.
What exactly are you talking about? Simone hasn't published any code so far (at least not to my knowledge). The attribute stuff is explicitely marked as not being meant to be used in the final implentation but to show how attributes could work (sidenote: attributes is a concept we've been talking about since CRUXCon 2004).
The only person who has so far posted code and suggested it to be used for future pkgutils was you (Jay) [1], and it hardly qualifies as an implementation in beta stages since it was just a c header file.
Did you misread me, or are you trying to twist my point? Mark said "setup something for public eyes" to which I replied "I tried that - it didn't fly." The fact that the header file I posted was a bare skeleton and more of a design document was exactly my point. I did not feel very strongly that we should even use it - it was simply an optional starting point for a group effort. Your response, however, was "that's not appropriate, we should start with discussion and planning" (which was exactly what that file was supposed to prompt). Lo and behold, you and Simone have both been working on rewrites in private. Do you see my point?
When I wanted to discuss it and dared to question your approach to use a C header as starting point for a _design_ discussion, you got offended ("..forget i posted it. you do it.") and figured it would be a good idea to attack my coding efforts ("but i've seen enough of your code to say with confidence that you do not always execute planning strategy").
Now this... what are you trying to achieve? Confused, Johannes
I'm struggling with my wording here. Having worked with you for some time on this project, I am under the impression that if you do not do something, it is not good enough for you to use. For that reason, I am frustrated and disinclined to participate anymore. I don't really want to go into detail about all of this on the list - maybe you and I can talk about it sometime if you want. But for now, I'm withdrawing from this discussion and future CRUX development. I will still maintain my ports and update them on crux.nu unless the group concludes that would be a liability. -- Jay Dolan jdolan.dyndns.org A: Because it's annoying to read. Q: Why is top-posting bad? A: Top posting. Q: What's the most annoying thing about usenet?
Hi, [ Now, this is pretty hard for me, since you're much more eloquent... ] On Wed, Apr 26, 2006 at 08:18:11 -0400, Jay Dolan wrote: [...]
What exactly are you talking about? Simone hasn't published any code so far (at least not to my knowledge). The attribute stuff is explicitely marked as not being meant to be used in the final implentation but to show how attributes could work (sidenote: attributes is a concept we've been talking about since CRUXCon 2004). [...] Did you misread me, or are you trying to twist my point? Mark said "setup something for public eyes" to which I replied "I tried that - it didn't fly." The fact that the header file I posted was a bare skeleton and more of a design document was exactly my point. I did not feel very strongly that we should even use it - it was simply an optional starting point for a group effort. Your response, however, was "that's not appropriate, we should start with discussion and planning" (which was exactly what that file was supposed to prompt). Which is exactly what the wiki page is supposed to prompt too. This whole thread was meant to get that discussion going. Is a public discussion and planning before starting to code so bad?
Lo and behold, you and Simone have both been working on rewrites in private. Do you see my point? No, I fail to see it. Completely. I really can't comment on Simone's work and have no influence on his spare time activities, however the pkgutils-attr I wrote code is NOT meant to be used (and I've stated that everywhere I could). IOW, all we have right now is a wiki page where everyone from the crux team can add his ideas, and your reaction on the work I put into it is to acuse me of hijacking the [pkgutils] project to win recogniton :-(.
I'm struggling with my wording here. Having worked with you for some time on this project, I am under the impression that if you do not do something, it is not good enough for you to use. For that reason, I am frustrated and disinclined to participate anymore. Reading this really disappoints me. You make it sound like I didn't let anyone participate. I tried hard to make 'PkgutilsIdeas' balanced, and it contains lots of ideas from other people and cruxcon. You had write access to that wiki page since day one, and I announced the page on IRC
[...] the very day I started it.
I don't really want to go into detail about all of this on the list - maybe you and I can talk about it sometime if you want. Sounds like a good idea.
Regards, Johannes -- Johannes Winkelmann mailto:jw@smts.ch Zurich, Switzerland http://jw.smts.ch
On 04/26/06 14:18 Jay Dolan wrote:
Did you misread me, or are you trying to twist my point? Mark said "setup something for public eyes" to which I replied "I tried that - it didn't fly." The fact that the header file I posted was a bare skeleton and more of a design document was exactly my point. I did not feel very strongly that we should even use it - it was simply an optional starting point for a group effort. Your response, however, was "that's not appropriate, we should start with discussion and planning" (which was exactly what that file was supposed to prompt). Lo and behold, you and Simone have both been working on rewrites in private. Do you see my point?
Hmm...what I see is that Johannes just posted a prototype to back up and better illustrate the concepts in the wiki page he wrote. Regarding my rewrite, it's an activity I'm doing at work for a in-house semi-embedded project that only sports a C compiler[1]. It's not meant as a replacement or proposal but barely a port of pkgutils fpr my own needings, I've only brought it to attention in order to give my first impressions about choosing C instead of sh, as you can read in the related mail in this thread.
I'm struggling with my wording here. Having worked with you for some time on this project, I am under the impression that if you do not do something, it is not good enough for you to use. For that reason, I am frustrated and disinclined to participate anymore. I don't really want to go into detail about all of this on the list - maybe you and I can talk about it sometime if you want. But for now, I'm withdrawing from this discussion and future CRUX development. I will still maintain my ports and update them on crux.nu unless the group concludes that would be a liability.
I just hope you can resolve these little scratches and be on board again soon. Regards, Simone [1] http://fabrice.bellard.free.fr/tcc/ -- Simone Rota Bergamo, Italy - http://www.varlock.com
On Tue, Apr 25, 2006 at 07:56:15PM -0400, Jay Dolan wrote: [...]
But then how could one person hijack the project and win recognition? An attempt to form a baseline C design which was to be openly discussed from day one, rather than publicly previewed once in beta stages of implementation, was already put forth and immediately rejected.
sorry Jay, what's your point with such a statement ? As everyone can see, Johannes put a lot of work into ideas and proposals to keep the further development of CRUX up and running, that's exactly what we need after Per retiered from the project. I cannot see any comparable efforts from your side. I'm pretty sure that your statement dosn't encourage Johannes' motivation at all. Is that your intention ? quite annoyed Jürgen -- Juergen Daubert | mailto:jue@jue.li Korb, Germany | http://jue.li/crux
Juergen Daubert wrote:
sorry Jay, what's your point with such a statement ? As everyone can see, Johannes put a lot of work into ideas and proposals to keep the further development of CRUX up and running, that's exactly what we need after Per retiered from the project.
Yes, he certainly has.
I cannot see any comparable efforts from your side.
I'm pretty sure that your statement dosn't encourage Johannes' motivation at all. Is that your intention ?
I'd really like to /not/ reiterate myself. I think I was quite clear in my subsequent postings. I'm sorry if you dislike my opinion. I've said my piece, I let Johannes keep the last word on the issue, and at this point I'd really like to be left alone from this entire affair, alright? I like you, Jue. I enjoyed meeting you at CC '04, and I think you're a valuable asset to the project. My remarks were not aimed at you, and Johannes can defend himself just fine. -- Jay Dolan jdolan.dyndns.org A: Because it's annoying to read. Q: Why is top-posting bad? A: Top posting. Q: What's the most annoying thing about usenet?
Hi Simone, On Tue, Apr 25, 2006 at 13:49:11 +0200, Simone Rota wrote: [...] > - Independently from the features we'd like to push, I'm in > favour of a rewrite; I suggest either plain C with libtar > or sh, with portability in mind. When talking about a rewrite in C, do you mean a full rewrite, or just pkgadd and symlinks? > I'm in the middle of a ANSI C pkgutils port, (pkginfo, pkgrm almost > done), my first impressions while working on the code is that > using sh would significantly speed up developement and make it > easier to add features in the future. Especially for pkgmk, sh/bash feels more natural. > - I'm a bit worried about the metadata proposal: > - Not sure if leaving free, per-repository attributes can make things > easier rather than produce confusion / lack of standards Well, it's not exactly per-repository data (at least not planned to be that way). There are basically two categories of attributes which are defined by a port: - required attributes, like name, version, release and files - optional attributes like depends All from the first two are included in binary packages (and in Pkgfiles). The optional ones can be added to the package db by changing the code of pkgadd, so it's not meant to be adjusted by the user. The thing which makes it different than the previous situation (users could always adjust pkgadd and recompile if they wanted) is that we explicitely consider that i.e. uCRUX might use a fork of pkgutils with different set of attributes, which means that tools depending on optional attributes might only work there but not on regular CRUX or the other way around. However, the ports are perfectly the same; it's just the content of the package database that differs. Finally, obviously the configurability is not coupled with attributes per se, so even if we want to implement the attributes idea, we don't have to expose its extensibility that much :-). Thanks for your comments, Johannes -- Johannes Winkelmann mailto:jw@smts.ch Zurich, Switzerland http://jw.smts.ch
On 04/26/06 12:48 Johannes Winkelmann wrote: >> - Independently from the features we'd like to push, I'm in >> favour of a rewrite; I suggest either plain C with libtar >> or sh, with portability in mind. > When talking about a rewrite in C, do you mean a full rewrite, or just > pkgadd and symlinks? Mainly about pkadd and symlinks. Also I meant libarchive instead of libtar, sorry for the confusion. > Especially for pkgmk, sh/bash feels more natural. Yes, and that's why I won't ditch a sh-based pkgutils in which all tools are shell scripts. I havent't played with Han's reimplementation yet, it sounds an interesting approach. >> - I'm a bit worried about the metadata proposal: >> - Not sure if leaving free, per-repository attributes can make things >> easier rather than produce confusion / lack of standards > Well, it's not exactly per-repository data (at least not planned to be > that way). There are basically two categories of attributes which are > defined by a port: > - required attributes, like name, version, release and files > - optional attributes like depends > > The thing which makes it different than the previous situation (users > could always adjust pkgadd and recompile if they wanted) is that we > explicitely consider that i.e. uCRUX might use a fork of pkgutils with > different set of attributes, which means that tools depending on > optional attributes might only work there but not on regular CRUX or the > other way around. That's what I mean per-repository (your example CRUX / uCRUX fits well); IMHO it would be great to have a consistent package standard/metadata across all package providers, not because you're going to mix packages, but because additional tools / scripts dealing with packages can be reused independently from the package provider. We had a small example with the use of the dependencies within old contrib ports vs. base+opt; leaving out the fact that in that case we were dealing with ports, the not-so-standard depends line required the opt.dependencies addon. That said, if people would really benefit of more customizable pkgadd, I'm ok with it, I'm just trying to help get it right at 1st try ;) Regards, Simone -- Simone Rota Bergamo, Italy - http://www.varlock.com
(Seems things have cooled off, so let's get back on topic.) On Wed, 2006-04-26 at 19:19 +0200, Simone Rota wrote:
On 04/26/06 12:48 Johannes Winkelmann wrote:
- Independently from the features we'd like to push, I'm in favour of a rewrite; I suggest either plain C with libtar or sh, with portability in mind. When talking about a rewrite in C, do you mean a full rewrite, or just pkgadd and symlinks?
Mainly about pkadd and symlinks. Also I meant libarchive instead of libtar, sorry for the confusion.
I had a brief look at libarchive for a project some time ago, and while it's nice that you don't have to make your own gztype (saving ~40 LOC) I also found it to do a lot more than what I actually needed (which was to read gzipped tar archives, no more or less.) That said, the developer is an extremely nice guy and it has the advantage(?) of being actively maintained. Also, my (well, dpkg's) idea of encapsulating a tar.gz together with the meta data in a more appropriate archive format (ar or the like) would obviously benefit from libarchive's multi-format abilities.
Especially for pkgmk, sh/bash feels more natural.
Yes, and that's why I won't ditch a sh-based pkgutils in which all tools are shell scripts. I havent't played with Han's reimplementation yet, it sounds an interesting approach.
I dare to say that sh is the only sane "language" for pkgmk, perhaps with some extensions done in perl or similar. The key here is to get the levels right. pkgadd and pkgrm are crucial tools on _any_ CRUX system, so they should be as robust and independent as possible. I bet this is one of the main reasons why Per wanted to drop the libstdc++ dependency.
That said, if people would really benefit of more customizable pkgadd, I'm ok with it, I'm just trying to help get it right at 1st try ;)
Nobody ever does, but I share your enthusiasm :)
Originally Johannes Winkelmann wrote:
Hi there, Hello, dear devs!
Regarding PkgutilsIdeas:
[...]
Comments both on the wiki pages and the rest of this mail are highly welcome.
I've read all wiki pages, and I'm highly interested in package format of future CRUX packages. Well, I had couple of ideas recently about creating some sort of package format for CRUX. The goals to achieve, where: 1) simplicity 2) compatibility with existing pkgutils as much as possible 3) optionality 4) ability to install CRUX over the net entirely from binary packages The mentioned metadata is proposed to store in magical /ATTRIBUTES directory within a package in some form of attribute sets. I'd like to suggest my own solution I've reached some time ago in my dreams :) Actually there is a problem for me to maintain a dozen of CRUX PCs, including regular updates and automated installs with further configuring. I know that pkg-get and pkgsync are great tools, their help is precious. So my idea: Within a package archive in /var/lib/pkg/<packagename> is stored the entire port the package was built from. This is fairly simple :) This is very optional as soon as it is easy to disable such metadata include. This is very compatible, since you can use current pkgadd/prt-get to handle such packages. This is very flexible: simply saying, within /var/lib/pkg the some sort of 'local' ports tree is created. This gives an ability to rebuild the system easly in case of crash without worrying about different ports versions. This allows 'binary installs', - as soon as Pkgfile with all headers is included in a package, some script can easly process this information. This allows to create a centralized storage of some vital or heavy to compile packages. Also this allows users to create their own binary storage in order to do such installs: once the package is compiled it can be used further on a different CRUX box or after reinstalling existing CRUX box :) This also gives the ability to rebuilt the package. The other feature is that this method does not require further maintanance since the package format is prograssing in steps of ports tree. And finally this gives a very little overhead on package size - as soon as footprints are useless in packages. An advantage of such method over special /ATTRIBUTES directory is obvious: the package doesn't need special handling. But introducing /ATTRIBUTES means almost all existing package-related stuff of CRUX need to be rewrited. One can handle the local ports three in /var/lib/pkg/ as he/she wishes - even to delete it without any danger to package system itself. These were my thoughts. I felt I need to share them with you. Anyway I also appreciate any comments on these thoughs and any health critics as well :) -- Hope this was interesting, Oleksiy
participants (6)
-
Jay Dolan -
Johannes Winkelmann -
Juergen Daubert -
Mark Rosenstand -
Oleksiy V. Khilkevich -
Simone Rota