qt6-webengine, qt6-base, and cups interdependency issue
Hi all, I use a docker snapshot environment to create accurate footprints for the ports in the df repo. Ports are built in isolation to ensure there is no parasitic interaction in dependencies. This has worked well for several years. It catches missing dependencies and generates footprints that may end up reporting NEW items but never MISSING items. This methodology is broken by this test in the qt6-webengine Pkgfile: if [[ ! -e /usr/lib/qt6/plugins/printsupport/libcupsprintersupport.so ]]; then printf '\e[31m%s\e[m\n' " opt/qt6-base is not build with opt/cups support, and opt/qt6-webeninge depends on it! Please make sure opt/cups is installed, then rebuild opt/qt6-base. After that, you can attempt to build opt/qt6-webengine again. " exit 1 fi Because qt6-base is never built when cups is installed since it is not a dependency listed in qt6-base the build of qt6-webengine always fails. This is not a problem in my desktop environment. I use cups so it is already installed when I build qt6-base. This kind of hidden requirement in qt6-webengine breaks my quality control methods. For now all I can do is maintain my own port of qt6-base with cups listed in the dependencies. However, I suggest that this kind of test in qt6-webengine violates the tried-and-true intent of the dependency list in a Pkgfile in subtle ways and precludes some automation tools. I understand that not everyone that wants qt6-base wants cups but almost everthing that uses qt6-webengine does else the Print function in apps like qutebrowser can print but can only save to disk. I don't think including cups in qt6-base is much of a hit. Anyone using qt6-*anything* is already taking a huge hit in resources. Just because cups is installed with qt6-base no one has to start the cups daemon unless they specifically choose to in the their rc.conf. Again, thanks to the dev team for their hard work providing the best distro around.o Any suggestions? Comments? -Daryl
Hello. I am a bit off-topic, but still it was ringing .., so sorry. Daryl F wrote in <b6aa3054-1fc6-1acd-cd01-aa9d34859837@prairieturtle.ca>: |Hi all, Concur. |I use a docker snapshot environment to create accurate footprints for the |ports in the df repo. Ports are built in isolation to ensure there is no That is cool. (I do not, only unshare(1) etc namespace with overlay mounts for "security" or whatever, etc. But possibly one day i will use /var/lib/pkg/db to modify the overlay mount, to be clean except for base and dependencies. Whatever "base" is, hm.) |parasitic interaction in dependencies. This has worked well for several |years. It catches missing dependencies and generates footprints that may |end up reporting NEW items but never MISSING items. | |This methodology is broken by this test in the qt6-webengine Pkgfile: I did not report it, but for some time llvm or what checks and uses clang for building when installed, which is doomed to fail on interdependency issues, unless *all* parties (llvm, compiler-rt, clang) are build and *then* installed, but not when each one is build then installed, as that chain will not survive. (Ie that striked me as i did not remove me in my overlay. Even more overlay work ... Prairieturtle is a funny domain name!) ... --steffen | |Der Kragenbaer, The moon bear, |der holt sich munter he cheerfully and one by one |einen nach dem anderen runter wa.ks himself off |(By Robert Gernhardt)
P.S.: i think it is no good to exclude things like especially openssl from 'Depends on:' i would hope for the now maintained prt things to test for missing dependencies for especially openssl. If one does a git grep -i 'depends o' | grep openssl | wc -l you see a chequered picture, even in core/, but i claim that only those which do are right. Ie i have forgotten what the policy is, but if it was "not needed if part of the basesystem" then this is no good, especially since not all programs are so loud and clear like postfix is: Apr 24 20:16:04 outwall/smtpd[1402]: warning: run-time library vs. compile-time header version mismatch: OpenSSL 3.3.0 may not be compatible with OpenSSL 3.2.0 (this from server running AlpineLinux). Ie "prt-get dependent X" should get you going to avoid such actual incompatibilities, in my humble opinion. So for example libarchive has OPTION(ENABLE_OPENSSL "Enable use of OpenSSL" ON) (for checksums and such, ie, one would really, really want this even) and links all the things, but will not be recompiled automatically. Ciao, and greetings from Germany, --steffen | |Der Kragenbaer, The moon bear, |der holt sich munter he cheerfully and one by one |einen nach dem anderen runter wa.ks himself off |(By Robert Gernhardt)
On Wed, 24 Apr 2024, Steffen Nurpmeso wrote:
Hello.
I am a bit off-topic, but still it was ringing .., so sorry.
Daryl F wrote in <b6aa3054-1fc6-1acd-cd01-aa9d34859837@prairieturtle.ca>: |Hi all,
Concur.
|I use a docker snapshot environment to create accurate footprints for the |ports in the df repo. Ports are built in isolation to ensure there is no
That is cool. (I do not, only unshare(1) etc namespace with overlay mounts for "security" or whatever, etc. But possibly one day i will use /var/lib/pkg/db to modify the overlay mount, to be clean except for base and dependencies. Whatever "base" is, hm.)
|parasitic interaction in dependencies. This has worked well for several |years. It catches missing dependencies and generates footprints that may |end up reporting NEW items but never MISSING items. | |This methodology is broken by this test in the qt6-webengine Pkgfile:
I did not report it, but for some time llvm or what checks and uses clang for building when installed, which is doomed to fail on interdependency issues, unless *all* parties (llvm, compiler-rt, clang) are build and *then* installed, but not when each one is build then installed, as that chain will not survive. (Ie that striked me as i did not remove me in my overlay. Even more overlay work ... Prairieturtle is a funny domain name!)
...
--steffen | |Der Kragenbaer, The moon bear, |der holt sich munter he cheerfully and one by one |einen nach dem anderen runter wa.ks himself off |(By Robert Gernhardt)
I live on the Canadian prairie. We kept Red-Eared Slider turtles inside for pets for years. Plus we have turtles living outside (tough buggers). Hence the domain name. -Daryl
Daryl F wrote in <f77623b7-b137-0cac-841b-71d84b4ea2fb@prairieturtle.ca>: |On Wed, 24 Apr 2024, Steffen Nurpmeso wrote: ... |I live on the Canadian prairie. We kept Red-Eared Slider turtles inside |for pets for years. Plus we have turtles living outside (tough buggers). Oh! I did, too! They were getting too large after a while (250 litre aquarium) so i brought them back to the pet store, and went back to fish. (Almost fourty years ago, though. .. I see they are even forbidden in the EU as of today -- what a mess, the giant hogweed is plain everywhere..) |Hence the domain name. I see. I should have used concrete autobahn jungle instead. |-Daryl --End of <f77623b7-b137-0cac-841b-71d84b4ea2fb@prairieturtle.ca> --steffen | |Der Kragenbaer, The moon bear, |der holt sich munter he cheerfully and one by one |einen nach dem anderen runter wa.ks himself off |(By Robert Gernhardt)
Hi all, Making cups a hard dependency of qt6-base (rather than "Optional" as it is now) would indeed be the easiest solution, as Daryl discovered by maintaining a duplicate port. Daryl F <wyatt@prairieturtle.ca> wrote:
However, I suggest that this kind of test in qt6-webengine violates the tried-and-true intent of the dependency list in a Pkgfile in subtle ways and precludes some automation tools.
The official prt-get still has no awareness of optional dependencies, almost 16 years after sepen first proposed a patch to make use of this field [1]. The fork I shared last year [2] has the logic to avoid the build error Daryl reported, if what's being tested is `prt-get depinst qt6-webengine` on a clean system (no qt6-base). But you have to put the line "softdeps yes" in the config file, or else pass --softdeps on the command line; otherwise the behavior of the original prt-get is retained. In the new mode, my patched prt-get first computes the minimal set of targets (which includes cups, if you ask for qt6-webengine), and then applies topological sorting with not just the hard dependency relationships, but also any optional dependencies among the targets. Feel free to try out my fork of prt-get [3], either for your personal use or for maintaining a published repo. If it eliminates the build errors reported by Daryl (and by hestia in the chatlog linked from [2]), without needing workarounds like a dup of qt6-base, then maybe we could get my patches incorporated into an official release. [1] https://lists.crux.nu/hyperkitty/list/crux-devel@lists.crux.nu/ thread/2QQLNMN3NEBOYMKP2QYOPVFPWZ4EVGAU/ [2] https://lists.crux.nu/hyperkitty/list/crux@lists.crux.nu/thread/ FKBNNVGKIPH5ORRUVWKQ3TIX36HRBDAV/ [3] https://git.crux.nu/farkuhar/prt-get/ -- John
On Wed, 24 Apr 2024, John McQuah wrote:
Hi all,
Making cups a hard dependency of qt6-base (rather than "Optional" as it is now) would indeed be the easiest solution, as Daryl discovered by maintaining a duplicate port.
Daryl F <wyatt@prairieturtle.ca> wrote:
However, I suggest that this kind of test in qt6-webengine violates the tried-and-true intent of the dependency list in a Pkgfile in subtle ways and precludes some automation tools.
The official prt-get still has no awareness of optional dependencies, almost 16 years after sepen first proposed a patch to make use of this field [1]. The fork I shared last year [2] has the logic to avoid the build error Daryl reported, if what's being tested is `prt-get depinst qt6-webengine` on a clean system (no qt6-base). But you have to put the line "softdeps yes" in the config file, or else pass --softdeps on the command line; otherwise the behavior of the original prt-get is retained. In the new mode, my patched prt-get first computes the minimal set of targets (which includes cups, if you ask for qt6-webengine), and then applies topological sorting with not just the hard dependency relationships, but also any optional dependencies among the targets.
Feel free to try out my fork of prt-get [3], either for your personal use or for maintaining a published repo. If it eliminates the build errors reported by Daryl (and by hestia in the chatlog linked from [2]), without needing workarounds like a dup of qt6-base, then maybe we could get my patches incorporated into an official release.
[1] https://lists.crux.nu/hyperkitty/list/crux-devel@lists.crux.nu/ thread/2QQLNMN3NEBOYMKP2QYOPVFPWZ4EVGAU/
[2] https://lists.crux.nu/hyperkitty/list/crux@lists.crux.nu/thread/ FKBNNVGKIPH5ORRUVWKQ3TIX36HRBDAV/
[3] https://git.crux.nu/farkuhar/prt-get/
-- John
Hi, softdeps in the docker environment would pollute the new footprints with things that the end consumer of one of my ports may not be using in their system. For me I consider a MISSING file at pkgmk time is fatal. I don't mind NEW because it usually means I've already installed something that autoconfigure or friends found and added support for. In my case the only solution is if qt6-base has a real dependency for cups not an optional dependency. I recall the discussion about optional dependencies. I was neither for or against it. Some might find it useful to automatically include them but I just review the ports and manually install an optionals I like. -Daryl
Daryl F wrote in <b3b765a6-1ff7-2a75-58d9-84c03b1455fb@prairieturtle.ca>: |On Wed, 24 Apr 2024, John McQuah wrote: ... |system. For me I consider a MISSING file at pkgmk time is fatal. I don't |mind NEW because it usually means I've already installed something that |autoconfigure or friends found and added support for. I concur a hundred percent. ... |I recall the discussion about optional dependencies. I was neither for or |against it. Some might find it useful to automatically include them but I |just review the ports and manually install an optionals I like. Yes. An option would possibly be a syntax like the VERP system of SMTP / mailing-lists, ie, you give the optionals as part of a name, and the new prt-get would fail if the mentioned optionals would not be mentioned in the Pkgfile, but otherwise do the necessary steps. Eg the core port ninja (unfortunately) has # Optional: bash-completion zsh so you could install ninja,zsh or ninja,zsh,bash-completion or ninja,bash-completion,zsh, and the new prt-get would note the used optionals in the DB, too. So then prt-get update ninja would effectively mean what was used. And if the Pkgfile had changed in the meantime and zsh would no longer be Optional:, you had to uninstall first to get that updated. Packaging systems are very, very complicated beasts. Anyway i hope that "prt-get isinst" is anything necessary in the runnable code of Pkgfile's. Good night from Germany, --steffen | |Der Kragenbaer, The moon bear, |der holt sich munter he cheerfully and one by one |einen nach dem anderen runter wa.ks himself off |(By Robert Gernhardt)
Hi all, Steffen Nurpmeso <steffen@sdaoden.eu> wrote:
An option would possibly be a syntax like the VERP system of SMTP / mailing-lists, ie, you give the optionals as part of a name, and the new prt-get would fail if the mentioned optionals would not be mentioned in the Pkgfile, but otherwise do the necessary steps.
Following up on the example I used in my last reply, by providing the optional dependencies as separate command-line targets (not appendages to the main target tacked on by commas), you get all the benefits that Steffen is proposing, but without automatically rejecting the command as invalid if one of the additional targets is not explicitly mentioned as an optional dependency. In the latter case, the absence of any optional dependency relationship just gives the topological sorting algorithm more freedom in how it orders the targets. Here's the partial output of `prt-get depinst --test --softdeps qt6-webengine numactl pipewire` on one system (trimmed for readability): *** prt-get: test mode -- Successful packages minizip libb2 abseil-cpp snappy re2 numactl xorg-xcb-util-cursor qt6-base qt6-shadertools qt6-serialport qt6-declarative qt6-positioning qt6-tools qt6-websockets qt6-webchannel pipewire qt6-webengine prt-get: install successful. *** prt-get: test mode end
Eg the core port ninja (unfortunately) has
# Optional: bash-completion zsh
so you could install ninja,zsh or ninja,zsh,bash-completion or ninja,bash-completion,zsh, and the new prt-get would note the used optionals in the DB, too.
Modifying the DB format, to indicate the precise variant of a port that was installed, is veering too far from KISS for my tastes. Proceeding in that direction will take you to the Nix packaging system, with thousands of possible variants in the DB that will require more parsing logic than our tools currently have. I preferred to retain as much of the CRUX simplicity as possible when patching prt-get to support optional deps.
So then prt-get update ninja would effectively mean what was used. And if the Pkgfile had changed in the meantime and zsh would no longer be Optional:, you had to uninstall first to get that updated.
I have two different branches in my forked prt-get repo, one that merges install/update/depinst so that dependencies are always resolved and all targets in the minimal set of ports are brought up to date (mixed-upinst), and another branch that retains the traditional distinction among install/update/depinst. If you gave Steffen's example `prt-get update ninja` to the mixed-upinst variant, then ninja would be updated after all of its hard dependencies that are not the current version, and also after all of its currently-installed soft dependencies that are not the current version [1]. Now suppose the maintainer moves zsh from "Optional" to "Depends on", and the user did not yet have zsh installed. In that case, the mixed-upinst variant is smart enough to add zsh as an install target, and build zsh before trying to update ninja. Or suppose the maintainer decides to delete all zsh references from the ninja Pkgfile. Then the mixed-upinst variant would happily leave zsh untouched, updating only the other targets in the minimal set (ninja and whatever else is out of date in its remaining dependency tree). And if no other installed port lists zsh as a hard or soft dependency, then pkgfoster will show zsh as a possible target for removal, thanks to the softdeps logic in prt-get listorphans. If you gave Steffen's example to the prt-get in my "master" branch, it would do the usual thing and just update the one target passed on the command line (ninja). Any other outdated ports in the dependency tree would be left at their current versions. To update all ports that are used by ninja you would have to do something convoluted like prt-get update $(prt-get depends --softdeps ninja | awk '/^\[u/ {print $2}'). Or even easier: prt-get sysup --softdeps (if you can afford to wait for all the other outdated ports on your system to be rebuilt). Footnote: [1] Thankfully both of ninja's optional dependencies still use autotools as their build system (not ninja itself), so the check for a dependency cycle is not triggered. If following softdeps would create a dependency cycle, then the offending edge of the digraph is silently ignored, and prt-get reverts to the default behavior of respecting only hard dependencies. Cheers, John
Hello. John McQuah wrote in <3REIYV929TH6I.3S3P20GMVHVLB@rawtext.club>: |Steffen Nurpmeso <steffen@sdaoden.eu> wrote: |> An option would possibly be a syntax like the VERP system of SMTP |> / mailing-lists, ie, you give the optionals as part of a name, and |> the new prt-get would fail if the mentioned optionals would not be |> mentioned in the Pkgfile, but otherwise do the necessary steps. | |Following up on the example I used in my last reply, by providing |the optional dependencies as separate command-line targets (not |appendages to the main target tacked on by commas), you get all the |benefits that Steffen is proposing, but without automatically rejecting |the command as invalid if one of the additional targets is not |explicitly mentioned as an optional dependency. In the latter case, the That does not make sense, no? I do not want to install them by themselves, i only install them as optional dependencies for another thing. |absence of any optional dependency relationship just gives the |topological sorting algorithm more freedom in how it orders the targets. |Here's the partial output of `prt-get depinst --test --softdeps |qt6-webengine numactl pipewire` on one system (trimmed for readability): ... Yeah. That was not what i meant. I personally surely will never use a --softdeps that simply installs *none or all* optional dependencies. I mean, sure, that is then at least an official way to trigger use of the optional things, but all or nothing? For example, look at qemu-all: # Optional: alsa-lib fuse3 gnutls libaio libjpeg-turbo libpng libsdl2 libseccomp libslirp libssh liburing libxkbcommon nettle nfs-utils numactl pipewire pulseaudio python3-sphinx python3-sphinx_rtd_theme sdl2_image snappy xkeyboard-config xkeyboard-config I miss gtk somehow, hmm, but anyway, does pipewire *and* pulseaudio make any sense at all? (I only alsa.) I know i have libsdl2 (for those installers which i just cannot bend to go ncurses, and only as long as i can ssh into the machine, then of course: "none"), but none of the other sdl2 things. And surely would not want to have all that mess! ... |> Eg the core port ninja (unfortunately) has |> |> # Optional: bash-completion zsh |> |> so you could install ninja,zsh or ninja,zsh,bash-completion or or better maybe ninja/zsh. |> ninja,bash-completion,zsh, and the new prt-get would note the |> used optionals in the DB, too. | |Modifying the DB format, to indicate the precise variant of a port that |was installed, is veering too far from KISS for my tastes. Proceeding in Well there could be another file in the same dir, OpenBSD calls these things then, hm, flavours i think. flavour.db? |that direction will take you to the Nix packaging system, with thousands |of possible variants in the DB that will require more parsing logic than |our tools currently have. I preferred to retain as much of the CRUX |simplicity as possible when patching prt-get to support optional deps. Hm, maybe not if you split at the separator and sort the list. There is a main port, and its optional dependencies in sort order. |> So then prt-get update ninja would effectively mean what was used. |> And if the Pkgfile had changed in the meantime and zsh would no |> longer be Optional:, you had to uninstall first to get that |> updated. | |I have two different branches in my forked prt-get repo, one that |merges install/update/depinst so that dependencies are always resolved |and all targets in the minimal set of ports are brought up to date |(mixed-upinst), and another branch that retains the traditional |distinction among install/update/depinst. If you gave Steffen's example |`prt-get update ninja` to the mixed-upinst variant, then ninja would be |updated after all of its hard dependencies that are not the current |version, and also after all of its currently-installed soft dependencies |that are not the current version [1]. I would hope for that "prt-get quickdep" gains an option to include those (actually installed) optional dependencies. (I use that for my port-up.sh script.) |Now suppose the maintainer moves zsh from "Optional" to "Depends |on", and the user did not yet have zsh installed. In that case, the |mixed-upinst variant is smart enough to add zsh as an install target, |and build zsh before trying to update ninja. Isn't that just the usual dependency unfolding then? What is new with this? |Or suppose the maintainer decides to delete all zsh references from the |ninja Pkgfile. Then the mixed-upinst variant would happily leave zsh |untouched, updating only the other targets in the minimal set (ninja and |whatever else is out of date in its remaining dependency tree). And if That could then be covered by the above mechanism. Ie looking into the flavour.db first sees the "soft-dependency" of zsh, and no mention of zsh in the full DB, which then logically means zsh is a stale thing than can be removed? |no other installed port lists zsh as a hard or soft dependency, then |pkgfoster will show zsh as a possible target for removal, thanks to the |softdeps logic in prt-get listorphans. I want to point out that these commands are totally useless as long as the hard dependency lines are as messy (and in part even by project policy) as they are. Ie listorphans gives signify and tar and such here. And pkgfoster is a total no-brainer (sorry), i need acpid and alsa-lib, #?0|kent:crux-ports.git$ pkgfoster /usr/bin/doas (Re-)checking packages for orphans... Can't open cache file: /var/lib/pkg/prt-get.cache Can't open cache file: /var/lib/pkg/prt-get.cache Package 'acpid' not found Uninstall acpid? (y/N) n Can't open cache file: /var/lib/pkg/prt-get.cache Package 'alsa-lib' not found Uninstall alsa-lib? (y/N) ^C Not that i care, i have a port-trim.sh which does it right (but relies on git-based ports to be able to deal with packages which were deliberately not updated aka are not up-to-date; but, then). |If you gave Steffen's example to the prt-get in my "master" branch, it |would do the usual thing and just update the one target passed on the |command line (ninja). Any other outdated ports in the dependency tree |would be left at their current versions. To update all ports that are |used by ninja you would have to do something convoluted like prt-get |update $(prt-get depends --softdeps ninja | awk '/^\[u/ {print $2}'). Or |even easier: prt-get sysup --softdeps (if you can afford to wait for all |the other outdated ports on your system to be rebuilt). To be honest, except for "quickdep --softdeps" this entire concept does not meet my needs. "All or nothing" for softdeps, this will not work out as such for many even, at least in a sane way. (Ie multiple graphical interfaces for qemu, multiple database backends for some server, etc etc.) I would rather see the really thoughtful "local patch mechanism" that one list member had developed, as it keeps signatures working but still allows local adjustments. |Footnote: | |[1] Thankfully both of ninja's optional dependencies still use autotools |as their build system (not ninja itself), so the check for a dependency |cycle is not triggered. If following softdeps would create a dependency |cycle, then the offending edge of the digraph is silently ignored, and |prt-get reverts to the default behavior of respecting only hard |dependencies. Ciao, --steffen | |Der Kragenbaer, The moon bear, |der holt sich munter he cheerfully and one by one |einen nach dem anderen runter wa.ks himself off |(By Robert Gernhardt)
Steffen Nurpmeso wrote in <20240425190115.qFKCR8uM@steffen%sdaoden.eu>: |John McQuah wrote in | <3REIYV929TH6I.3S3P20GMVHVLB@rawtext.club>: ||Steffen Nurpmeso <steffen@sdaoden.eu> wrote: ||> An option would possibly be a syntax like the VERP system of SMTP ||> / mailing-lists, ie, you give the optionals as part of a name, and ||> the new prt-get would fail if the mentioned optionals would not be ||> mentioned in the Pkgfile, but otherwise do the necessary steps. || ||Following up on the example I used in my last reply, by providing ||the optional dependencies as separate command-line targets (not ||appendages to the main target tacked on by commas), you get all the ||benefits that Steffen is proposing, but without automatically rejecting ||the command as invalid if one of the additional targets is not ||explicitly mentioned as an optional dependency. In the latter case, the | |That does not make sense, no? I do not want to install them by |themselves, i only install them as optional dependencies for |another thing. That was of course a very unfair statement given that the situation of today is the very same: flat hierarchy etc. I truly love the rename= thing, and i truly love that i can script something with the sh(1)ell around all that (even though quite hacky with sourcing of pkgmk.conf and per-port Pkgfile, etc), and it works out just fine for me, port-up -qd;port-up -qn and you are done. I have no opinion on your --softdeps thing actually, apart from that "none or all" is surely nothing i want for approaching 100 percent of all cases. And i could imagine that adding say "flavour" lines, aka "several optional dependency groups", is likely too much effort for CRUX. (Especially given that the "real CRUX team" is very small and handles many packages.) I find myself -- i have a /root/hosts/HOSTNAME/crux.pkg-listinst file -- that sometimes i do not understand why i have a port, and then it is *very* hard to find why it is there; as somethimes it is a dynamically loaded module, say for example within /usr/lib/ImageMagick-7.1.1/modules-Q16HDRI/coders/, that requires that port, and, as far as i know, CRUX offers *zero* hands to do this; revdep does not dig that, unfortunately. Well, i do not know. Pkg stuff is very, *very* hard. Ciao, --steffen | |Der Kragenbaer, The moon bear, |der holt sich munter he cheerfully and one by one |einen nach dem anderen runter wa.ks himself off |(By Robert Gernhardt)
Hi Daryl, Daryl F <wyatt@prairieturtle.ca> wrote:
softdeps in the docker environment would pollute the new footprints with things that the end consumer of one of my ports may not be using in their system.
I might not have the clearest picture of how your docker setup works, so please correct me if this description is wrong: when generating the footprint of a new port (take qutebrowser as an example), you load a docker image that consists of a core-only environment, and run `prt-get depinst qutebrowser`. Along the way, it might generate footprints for some dependencies that aren't yet installed (those that didn't have a footprint already), but the only footprint you keep for the purposes of publishing the qutebrowser port is the qutebrowser footprint. In other words, the footprints of dependencies are NOT updated, even if pkgmk reports NEW files for some of them. If that's how your docker setup works, I don't expect pollution of the resulting footprint by toggling the --softdeps switch. As I wrote, it still calculates the _minimal_ set of dependencies needed to install the requested targets, using hard dependencies just as the original prt-get does. The only effect of --softdeps is in the topological sorting: in the event that there happen to be optional dependency relationships among the resulting minimal set of targets, prt-get will be smart enough to recognize them and move the optional dependencies earlier in the queue (e.g., cups before qt6-base, resulting in NEW files in the qt6-base footprint, but nothing NEW in qt6-webengine which is the actual target requested for installation). This feature is similar to what Steffen described as "VERP system of SMTP / mailing lists" (but the optional dependencies appear as separate arguments, not adornments of the main target tacked on with commas). By way of illustration, here's the output of `prt-get depinst --test --softdeps qt6-webengine` on a working system (not core-only, but at least it lacks cups and qt6 stuff): *** prt-get: test mode -- Packages installed minizip libb2 abseil-cpp snappy re2 python3-six python3-webencodings python3-html5lib cups xorg-xcb-util-cursor qt6-base qt6-shadertools qt6-serialport qt6-declarative qt6-positioning qt6-tools qt6-websockets qt6-webchannel qt6-webengine -- installed packages with README files: /usr/ports/opt/cups prt-get: installed successfully Note that although qt6-webengine lists numactl and pipewire as optional dependencies, and the test system did not have these ports installed, the prt-get command did not add them to the set of install targets. If you want a maximal port, you have to do more work to get it (see link at the end of this post). Topological sorting is also modified in a sysup operation: my patched prt-get has the logic to update an optional dependency before the port that optionally depends on it (e.g., vulkan-loader before mesa, which is not the order you're guaranteed to get with the original prt-get). In fact, it was the optional dependencies of mesa that originally prompted my work on this fork.
I recall the discussion about optional dependencies. I was neither for or against it. Some might find it useful to automatically include them but I just review the ports and manually install an optionals I like.
I hope I've given a better explanation of what "softdeps yes" in the configuration file (or --softdeps on the command line) actually does. To reiterate: it does NOT automatically include as install targets all the ports in the "Optional:" field, it only applies a smarter sorting algorithm to the ports that are passed as arguments (and their hard dependencies). The expanded man-page demonstrates an unwieldy command if you want to build a "maximally interdependent" package, but this is not the default behavior when you set "softdeps yes". https://git.crux.nu/farkuhar/prt-get/src/branch/master/doc/prt-get.8#L719 -- John
On Thu, 25 Apr 2024, John McQuah wrote:
Hi Daryl,
Daryl F <wyatt@prairieturtle.ca> wrote:
softdeps in the docker environment would pollute the new footprints with things that the end consumer of one of my ports may not be using in their system.
I might not have the clearest picture of how your docker setup works, so please correct me if this description is wrong: when generating the footprint of a new port (take qutebrowser as an example), you load a docker image that consists of a core-only environment, and run `prt-get depinst qutebrowser`. Along the way, it might generate footprints for some dependencies that aren't yet installed (those that didn't have a footprint already), but the only footprint you keep for the purposes ofi publishing the qutebrowser port is the qutebrowser footprint. In other words, the footprints of dependencies are NOT updated, even if pkgmk reports NEW files for some of them.
prt-get is not used in the docker environment only pkgmk, pkgadd and pkgrm. You are correct the build process begins with a docker snapshot of the core-only environment. Package tarballs from previous builds are retained in an read/write overlay. 1) A list of dependencies for the target port is developed using prt-get quickdep. All the dependencies found by quickdep are also resolved using quickdep until all the dependencies are discovered. This include any accidental core ports but step 2) below will find them already installed. 2) If a dependency is not installed a pkgadd is attempted from the read/write tarballs. If that fails all previously installed dependencies are pgkrm'd. The missing dependency becomes the target port and the process recurses to step 1). If the pkgadd works then the next dependency is examined at step 2) until all dependencies are installed. When all dependencies are installed advance to the next step. 3) The target port is built with pkgmk. Only that footprint is retained. The tarball created is retained in the read/write overlay. 4) Remove all packages installed in step 2) Each time a build is started it begins with the core-only docker image. All of the tarballs from pkgmk are retained for future use since they are each "pure" ie. having no optional or autoconfigure induced differences during their build. In step 2) the stored tarball's version is examined. If it is out-of-date then it is considered a failure and the recursion is begun so that the current version is used. Periodically a script is run that examines all the stored tarballs. Any that are out-of-date are rebuilt with the above process. The above method wears out disks. The first time qt6-webengine is built, before there are any stored tarballs, many are loaded and then unloaded so that a missing dependency can be built without influence from other ports. But it works flawlessly not matter what autoconf, cmake, etc. decide to due. Unneeded software can not be noticed by them at build time. Weekly a ports -u is done and the docker core-only environment is rebuilt.
If that's how your docker setup works, I don't expect pollution of the resulting footprint by toggling the --softdeps switch.As I wrote, it still calculates the _minimal_ set of dependencies needed to install the requested targets, using hard dependencies just as the original prt-get does. The only effect of --softdeps is in the topological sorting: in the event that there happen to be optional dependency relationships among the resulting minimal set of targets, prt-get will be smart enough to recognize them and move the optional dependencies earlier in the queue (e.g., cups before qt6-base, resulting in NEW files in the qt6-base footprint, but nothing NEW in qt6-webengine which is the actual target requested for installation). This feature is similar to what Steffen described as "VERP system of SMTP / mailing lists" (but the optional dependencies appear as separate arguments, not adornments of the main target tacked on with commas).
This is a really cool solution and would result in qt6-base being built with cups support as needed by qt6-webengine. However, the footprint could still be polluted occasionally but is not prt-get's fault. It is due to autoconfigure, cmake, etc. and occasionaly the logic added in Pkgfiles that changes the configure options are installed. The kerberos check in qt6-webengine's Pkgfile for instance. More on this below.
By way of illustration, here's the output of `prt-get depinst --test --softdeps qt6-webengine` on a working system (not core-only, but at least it lacks cups and qt6 stuff):
*** prt-get: test mode
-- Packages installed minizip libb2 abseil-cpp snappy re2 python3-six python3-webencodings python3-html5lib cups xorg-xcb-util-cursor qt6-base qt6-shadertools qt6-serialport qt6-declarative qt6-positioning qt6-tools qt6-websockets qt6-webchannel qt6-webengine
-- installed packages with README files: /usr/ports/opt/cups
prt-get: installed successfully
As a contrived example: in the above abseil-cpp is dependency of re2 and is needed at build time. It is left installed and then cups is built. Pretend the cups autoconfigure script detects that abseil-cpp is installed and builds with extra features (files) and is installed. When qt6-webengine is built it detects those extra files and it too builds some extra feature (files) and installs those too. Fast forward into the and pretend that a hard dep for cups was added to minizip (crazy I know but it wants to print errors now). This time cups is built before re2 so abseil-cpp is not built so cups omits those files related to it being present. Now the footprint for qt6-webengine is corrupt. The extra files were not built by cups so qt6-webengine did not build its cups-related files either. Any depinst of qt6-webengine on a new CRUX will fail with MISSING files. This happened enough times when I used a VM and prt-get as my "clean" test system that I developed the docker method. In all cases leaving anything installed in the test environment will result in autoconfigure and friends detecting something and (not so) helpfully add extra files to their resulting tarballs possibly affecting other autoconfigure scripts downstream.
Note that although qt6-webengine lists numactl and pipewire as optional dependencies, and the test system did not have these ports installed, the prt-get command did not add them to the set of install targets. If you want a maximal port, you have to do more work to get it (see link at the end of this post).
Topological sorting is also modified in a sysup operation: my patched prt-get has the logic to update an optional dependency before the port that optionally depends on it (e.g., vulkan-loader before mesa, which is not the order you're guaranteed to get with the original prt-get). In fact, it was the optional dependencies of mesa that originally prompted my work on this fork.
I recall the discussion about optional dependencies. I was neither for or against it. Some might find it useful to automatically include them but I just review the ports and manually install an optionals I like.
I hope I've given a better explanation of what "softdeps yes" in the configuration file (or --softdeps on the command line) actually does. To reiterate: it does NOT automatically include as install targets all the ports in the "Optional:" field, it only applies a smarter sorting algorithm to the ports that are passed as arguments (and their hard dependencies). The expanded man-page demonstrates an unwieldy command if you want to build a "maximally interdependent" package, but this is not the default behavior when you set "softdeps yes".
Yes, much clearer now.
https://git.crux.nu/farkuhar/prt-get/src/branch/master/doc/prt-get.8#L719
-- John
With the docker environment I use there is a complete disconnect between the building of qt6-base and qt6-webengine. When qt6-base is built there is no qt6-webengine available. And when qt6-webengine is built only the qt6-base that was built independently is installed. So at build time of qt6-base there is no way to know that the optional dependency for cups in qt6-base is a hard dependency in qt6-webengine. Since 2017 this hasn't been a problem but in the meantime we've seen more conditional tests added to Pkgfiles that alter the configuration behaviour. In the case of qt6-webengine it is a good test because it clearly shows what is wrong before a lengthy compile that will end in a failure. But a hard dep in qt6-base with cups would have fixed that and my automation's problem too. When I think of all the bitcoin I could have mined with the electricity from failed Qt compiles :) -Daryl
Hi Daryl, Daryl F <wyatt@prairieturtle.ca> wrote:
When qt6-base is built there is no qt6-webengine available. And when qt6-webengine is built only the qt6-base that was built independently is installed. So at build time of qt6-base there is no way to know that the optional dependency for cups in qt6-base is a hard dependency in qt6-webengine.
Thanks for explaining in detail how your docker build environment works. To the extent that it deviates from how most CRUX users will attempt to install their desktop environment, you can expect a struggle with the way our ports are currently designed. The contrib maintainers tend to write their Pkgfiles to accommodate a workflow based on `prt-get depinst $desired_toplevel_port`, not a workflow based on bootstrapping the individual dependencies "from the ground up." Hence the common idiom of conducting `prt-get isinst` or filesystem tests to determine whether the dependencies were configured appropriately.
All of the tarballs from pkgmk are retained for future use since they are each "pure" ie. having no optional or autoconfigure induced differences during their build.
Reusing a package tarball that was built in a clean docker image is already a departure from how the typical CRUX user will proceed, and therein lies your frustration with qt6-base and qt6-webengine. I sympathize with your motivation, and I can't imagine you'd be eager to adopt a workflow that disallows such reuse (opting instead to build each dependency from scratch in the clean docker image, as I naively envisioned in my first reply).
In the case of qt6-webengine it is a good test because it clearly shows what is wrong before a lengthy compile that will end in a failure. But a hard dep in qt6-base with cups would have fixed that and my automation's problem too.
The recent IRC rants against linux-pam are premised on the claim that CRUX is not living up to its name, and is now prioritizing convenience for maintainers rather than stripping away from core everything that's not essential. But at least in the case of "fluid Pkgfiles" like qt6-base, the spirit of stripping away the non-essential has been respected. You don't have to install cups to get a working build of qt6-base, just as you don't have to install linux-pam to get a usable tty on a CRUX system. Sure, opting for the hard dep in qt6-base makes things more convenient for other ports. But similar reasoning was invoked to justify putting linux-pam in core. So "fluid Pkgfiles" are arguably more aligned with the CRUX spirit of "stripping away the non-essential." Here's another solution that obviates "fluid Pkgfiles", but still accommodates the minimalist users who have no need for cups or qt6-webengine. Have qt6-webengine depend on a duplicate port qt6-base-cups, which is identical to qt6-base except that cups is a hard dependency. (It's similar to the way I configured inkscape to depend on the near-duplicate poppler-ink, to guard against build failures when poppler gets updated to a version that inkscape does not yet support. While it does clutter /usr/lib with possibly-redundant files, at least the Pkgfile does not require any branching logic to ensure a successful build--- e.g. installing a customized poppler when the wrong soversion is found.) Then you can write in prt-get.aliases the line "qt6-base-cups: qt6-base" to inform other ports that a dependency on qt6-base is satisfied if qt6-base-cups is installed. Turning to the points Steffen raised about my fork of prt-get ... Steffen Nurpmeso wrote in <20240425190115.qFKCR8uM@steffen%sdaoden.eu>:
That does not make sense, no? I do not want to install them by themselves, i only install them as optional dependencies for another thing.
Perhaps it would have been clearer if I used Steffen's example. Instead of following the SMTP / mailing list syntax (prt-get depinst ninja,zsh), just write `prt-get depinst ninja zsh`, and with "softdeps yes" in its configuration, my patched prt-get will recognize that zsh---as an optional dependency of ninja---should be moved earlier in the queue. In other words, topological sorting is performed with both types of edges in the digraph: edges connecting ports to their hard dependencies (what the original prt-get already considers), and edges connecting ports to the soft dependencies that are: - given on the command line, - already installed (in an update/sysup operation), OR - hard dependencies of other ports passed as arguments. Changing the format of the DB to indicate which "flavour" of ninja was installed is an example of "over-design". Once you allow such complexity into your DB, you might find it difficult to update ninja when the maintainer later decides to rework the dependencies. I prefer keeping the package database in its current simple format, rather than aiming for deterministic builds and completely-specificied dependencies a la NixPkgs.
I have no opinion on your --softdeps thing actually, apart from that "none or all" is surely nothing i want for approaching 100 percent of all cases.
The language "none or all" is a complete mischaracterization. You are free to inspect the Pkgfile and choose a proper subset of the optional dependencies, passing them as arguments to a `prt-get depinst` command. Using Steffen's example again: `prt-get depinst --softdeps ninja zsh` will install the optional dependency zsh, but NOT the optional dependency bash-completion (which was not given as an argument). Or to use the qemu-all example: `prt-get depinst --softdeps qemu-all libsdl2` will ensure that libsdl2 is built before qemu-all, but it will NOT automatically try to install any of the other softdeps (Steffen can retain his ALSA-only setup without worrying that pipewire and pulseaudio will be installed by this command).
Now suppose the maintainer moves zsh from "Optional" to "Depends on", and the user did not yet have zsh installed. In that case, the mixed-upinst variant is smart enough to add zsh as an install target, and build zsh before trying to update ninja.
Isn't that just the usual dependency unfolding then? What is new with this?
The new feature is only found in my mixed-upinst branch, which customizes pkgadd flags for each target (no flag if the target is not yet installed, -u if it's already installed and needs an update). Suppose your first installation of ninja did not have zsh as a hard dependency, and you never installed zsh for other reasons. But maybe the maintainer decides to make zsh a hard dependency in the next version. With the master branch of my repo, `prt-get update ninja` would refuse to inject zsh into the list of targets, because the -u flag would be invalid for the spawned pkgadd process. But the mixed-upinst branch will happily apply the -u flag only to targets already in the DB, and omit the -u flag if the target is new to the DB. There's no need to worry about the distinction between install and update, they both do the same thing in the mixed-upinst branch. As for dependency resolution (depinst versus install), that distinction has undergone a similar collapse: "install" and "update" resolve dependencies by default, but you can get the old "install" behaviour by passing the --nodeps flag. All these changes in the mixed-upinst branch were intended to make the most useful behaviour the default, so that new users don't get crucified for not reading the manual when they show up in the IRC channel and ask why they got a build failure with `prt-get install $some_port`.
prt-get listorphans.
I want to point out that these commands are totally useless as long as the hard dependency lines are as messy (and in part even by project policy) as they are. Ie listorphans gives signify and tar and such here.
I'd like to hear more discussion about the "project policy" that leads to messy "Depends on:" lines. It was my understanding that omitting runtime dependencies found in the core repo (unless they're compiled in) was intended to reduce clutter in the "Depends on" line. In his earlier message Steffen alluded to the widespread omission of openssl from the "Depends on" line (per project policy), resulting in runtime warnings from ports like postfix. If we start including even the core runtime dependencies that are not compiled in, wouldn't the "Depends on" lines get messier than they already are? I didn't patch prt-get to strip away core ports from the output of the listorphans command, because it's doing exactly what the man-page promises. Overlaying the additional knowledge of project policy that dictates how our "Depends on" lines are written is a job for an external tool. Given that a brief discussion is sometimes all it takes to change project policy (or a brief internal monologue to change the policy governing your private ports collection), a wrapper script in a port maintainer's workflow (like Steffen's port-trim.sh) is easier to adapt to new circumstances, rather than burying the consensus decision inside the subroutine for `prt-get listorphans`. My expanded man-page has a few examples that the new user can build upon, when writing such a wrapper script. https://git.crux.nu/farkuhar/prt-get/src/branch/master/doc/prt-get.8#L752 Cheers, John
Hi John On Fri, 26 Apr 2024, John McQuah wrote:
Hi Daryl,
Daryl F <wyatt@prairieturtle.ca> wrote:
When qt6-base is built there is no qt6-webengine available. And when qt6-webengine is built only the qt6-base that was built independently is installed. So at build time of qt6-base there is no way to know that the optional dependency for cups in qt6-base is a hard dependency in qt6-webengine.
Thanks for explaining in detail how your docker build environment works. To the extent that it deviates from how most CRUX users will attempt to install their desktop environment, you can expect a struggle with the way our ports are currently designed. The contrib maintainers tend to write their Pkgfiles to accommodate a workflow based on `prt-get depinst $desired_toplevel_port`, not a workflow based on bootstrapping the individual dependencies "from the ground up." Hence the common idiom of conducting `prt-get isinst` or filesystem tests to determine whether the dependencies were configured appropriately.
Understood. The docker environment is in no way meant to simulate anything a CRUX user might do. It is simply meant to built a smallest footprint with the least amount of uncontrolled interaction from the system environment. I count on the use prt-get depinst.
All of the tarballs from pkgmk are retained for future use since they are each "pure" ie. having no optional or autoconfigure induced differences during their build.
Reusing a package tarball that was built in a clean docker image is already a departure from how the typical CRUX user will proceed, and therein lies your frustration with qt6-base and qt6-webengine. I sympathize with your motivation, and I can't imagine you'd be eager to adopt a workflow that disallows such reuse (opting instead to build each dependency from scratch in the clean docker image, as I naively envisioned in my first reply).
I never use these tarballs anywhere but within the docker environment. All my instances (four) of CRUX use prt-get depinst/sysup.
In the case of qt6-webengine it is a good test because it clearly shows what is wrong before a lengthy compile that will end in a failure. But a hard dep in qt6-base with cups would have fixed that and my automation's problem too.
The recent IRC rants against linux-pam are premised on the claim that CRUX is not living up to its name, and is now prioritizing convenience for maintainers rather than stripping away from core everything that's not essential. But at least in the case of "fluid Pkgfiles" like qt6-base, the spirit of stripping away the non-essential has been respected. You don't have to install cups to get a working build of qt6-base, just as you don't have to install linux-pam to get a usable tty on a CRUX system. Sure, opting for the hard dep in qt6-base makes things more convenient for other ports. But similar reasoning was invoked to justify putting linux-pam in core. So "fluid Pkgfiles" are arguably more aligned with the CRUX spirit of "stripping away the non-essential."
Here's another solution that obviates "fluid Pkgfiles", but still accommodates the minimalist users who have no need for cups or qt6-webengine. Have qt6-webengine depend on a duplicate port qt6-base-cups, which is identical to qt6-base except that cups is a hard dependency. (It's similar to the way I configured inkscape to depend on the near-duplicate poppler-ink, to guard against build failures when poppler gets updated to a version that inkscape does not yet support. While it does clutter /usr/lib with possibly-redundant files, at least the Pkgfile does not require any branching logic to ensure a successful build--- e.g. installing a customized poppler when the wrong soversion is found.) Then you can write in prt-get.aliases the line "qt6-base-cups: qt6-base" to inform other ports that a dependency on qt6-base is satisfied if qt6-base-cups is installed.
That saves me no time. I can maintain my own port of qt6-base-cups with a hard dep for cups in it or maintain my own port of qt6-webengine. I've supported qt6-webengine before and it is a lot more work than qt6-base. Fortunately, over time, the opt port converged with mine and I was able to abandon it. On the topic of PAM: I don't care one way or the other. If it makes life for the CRUX team (bless them) easier then it should be in core. I disagree with any that say CRUX core should be the tiniest possible. AFAIK CRUX was never targeted at embedded systems. It has done very well at being a fine distro for those building servers and desktops. I don't think I've ever heard of distro that was good at that and embedded systems. Very different consumers IMO. I especially like that it has never gone to the overly complex places that Gentoo and Arch have. Arch PKGBUILD files are becoming difficult to translate to CRUX :) John, thanks for your time in this discussion. I have absolutely no complaints about what the CRUX dev team policies are. I haven't been able to provide much resource to help them and would like to see them have the least amount of work possible. I started with CRUX 2.5 in 2009 and have seen the turnover of dev personnel over the years. I don't want any of them to burn out. If they ain't having fun with CRUX they should, as a group, shrink the work as much as they see fit. Regards, Daryl
Hi! On 24.04.2024 14:17, Daryl F wrote:
Hi all,
I use a docker snapshot environment to create accurate footprints for the ports in the df repo. Ports are built in isolation to ensure there is no parasitic interaction in dependencies. This has worked well for several years. It catches missing dependencies and generates footprints that may end up reporting NEW items but never MISSING items.
This methodology is broken by this test in the qt6-webengine Pkgfile:
if [[ ! -e /usr/lib/qt6/plugins/printsupport/libcupsprintersupport.so ]]; then printf '\e[31m%s\e[m\n' " opt/qt6-base is not build with opt/cups support, and opt/qt6-webeninge depends on it! Please make sure opt/cups is installed, then rebuild opt/qt6-base. After that, you can attempt to build opt/qt6-webengine again. " exit 1 fi
Because qt6-base is never built when cups is installed since it is not a dependency listed in qt6-base the build of qt6-webengine always fails.
This is not a problem in my desktop environment. I use cups so it is already installed when I build qt6-base.
So far, so good! I think that's true for a lot of people.
This kind of hidden requirement in qt6-webengine breaks my quality control methods. For now all I can do is maintain my own port of qt6-base with cups listed in the dependencies.
True, I have to work around it with my tooling as well to account for this fact and it makes life certainly not easier.
However, I suggest that this kind of test in qt6-webengine violates the tried-and-true intent of the dependency list in a Pkgfile in subtle ways and precludes some automation tools.
I take myself as an example: not everybody wants cups. I don't even own a printer anymore and rarely print anything at all. So, on a machine that installs qt6-webeninge, I will have cups installed, but on machines that don't I am surely not installing cups. The logic is there to account for both use cases, and leads to a slimmer dependency chain for a default install.
I understand that not everyone that wants qt6-base wants cups but almost everthing that uses qt6-webengine does else the Print function in apps like qutebrowser can print but can only save to disk.
I don't think including cups in qt6-base is much of a hit. Anyone using qt6-*anything* is already taking a huge hit in resources. Just because cups is installed with qt6-base no one has to start the cups daemon unless they specifically choose to in the their rc.conf.
cups is simply not needed for qt6-base, therefor I put it down as an optional dependency. While it might not be a big hit, but its a design choice to try and make all the things not needed for basic install optional. So while it hurts our tooling, I am very much leaning towards leaving it as it is.
Again, thanks to the dev team for their hard work providing the best distro around.o
Any suggestions? Comments?
Thanks for sharing your mind and again, sorry for having to wait for an answer. I hope that you can find a solution for your builds in the docker container, usually I run checks in my script that figures out if my buildcontainer is setup correctly, otherwise it installs/rebuilds ports, e.g. for qt6-webengine it will install xkeyboard-config, rebuild libxkbcommon, then installs cups and rebuilds qt6-base to finally build qt6-webengine. :-)
-Daryl
Best regards, Tim
participants (4)
-
Daryl F -
John McQuah -
Steffen Nurpmeso -
Tim Biermann