On Thu, 25 Apr 2024, John McQuah wrote:
Hi Daryl,
Daryl F <wyatt@prairieturtle.ca> wrote:
softdeps in the docker environment would pollute the new footprints with things that the end consumer of one of my ports may not be using in their system.
I might not have the clearest picture of how your docker setup works, so please correct me if this description is wrong: when generating the footprint of a new port (take qutebrowser as an example), you load a docker image that consists of a core-only environment, and run `prt-get depinst qutebrowser`. Along the way, it might generate footprints for some dependencies that aren't yet installed (those that didn't have a footprint already), but the only footprint you keep for the purposes ofi publishing the qutebrowser port is the qutebrowser footprint. In other words, the footprints of dependencies are NOT updated, even if pkgmk reports NEW files for some of them.
prt-get is not used in the docker environment only pkgmk, pkgadd and pkgrm. You are correct the build process begins with a docker snapshot of the core-only environment. Package tarballs from previous builds are retained in an read/write overlay. 1) A list of dependencies for the target port is developed using prt-get quickdep. All the dependencies found by quickdep are also resolved using quickdep until all the dependencies are discovered. This include any accidental core ports but step 2) below will find them already installed. 2) If a dependency is not installed a pkgadd is attempted from the read/write tarballs. If that fails all previously installed dependencies are pgkrm'd. The missing dependency becomes the target port and the process recurses to step 1). If the pkgadd works then the next dependency is examined at step 2) until all dependencies are installed. When all dependencies are installed advance to the next step. 3) The target port is built with pkgmk. Only that footprint is retained. The tarball created is retained in the read/write overlay. 4) Remove all packages installed in step 2) Each time a build is started it begins with the core-only docker image. All of the tarballs from pkgmk are retained for future use since they are each "pure" ie. having no optional or autoconfigure induced differences during their build. In step 2) the stored tarball's version is examined. If it is out-of-date then it is considered a failure and the recursion is begun so that the current version is used. Periodically a script is run that examines all the stored tarballs. Any that are out-of-date are rebuilt with the above process. The above method wears out disks. The first time qt6-webengine is built, before there are any stored tarballs, many are loaded and then unloaded so that a missing dependency can be built without influence from other ports. But it works flawlessly not matter what autoconf, cmake, etc. decide to due. Unneeded software can not be noticed by them at build time. Weekly a ports -u is done and the docker core-only environment is rebuilt.
If that's how your docker setup works, I don't expect pollution of the resulting footprint by toggling the --softdeps switch.As I wrote, it still calculates the _minimal_ set of dependencies needed to install the requested targets, using hard dependencies just as the original prt-get does. The only effect of --softdeps is in the topological sorting: in the event that there happen to be optional dependency relationships among the resulting minimal set of targets, prt-get will be smart enough to recognize them and move the optional dependencies earlier in the queue (e.g., cups before qt6-base, resulting in NEW files in the qt6-base footprint, but nothing NEW in qt6-webengine which is the actual target requested for installation). This feature is similar to what Steffen described as "VERP system of SMTP / mailing lists" (but the optional dependencies appear as separate arguments, not adornments of the main target tacked on with commas).
This is a really cool solution and would result in qt6-base being built with cups support as needed by qt6-webengine. However, the footprint could still be polluted occasionally but is not prt-get's fault. It is due to autoconfigure, cmake, etc. and occasionaly the logic added in Pkgfiles that changes the configure options are installed. The kerberos check in qt6-webengine's Pkgfile for instance. More on this below.
By way of illustration, here's the output of `prt-get depinst --test --softdeps qt6-webengine` on a working system (not core-only, but at least it lacks cups and qt6 stuff):
*** prt-get: test mode
-- Packages installed minizip libb2 abseil-cpp snappy re2 python3-six python3-webencodings python3-html5lib cups xorg-xcb-util-cursor qt6-base qt6-shadertools qt6-serialport qt6-declarative qt6-positioning qt6-tools qt6-websockets qt6-webchannel qt6-webengine
-- installed packages with README files: /usr/ports/opt/cups
prt-get: installed successfully
As a contrived example: in the above abseil-cpp is dependency of re2 and is needed at build time. It is left installed and then cups is built. Pretend the cups autoconfigure script detects that abseil-cpp is installed and builds with extra features (files) and is installed. When qt6-webengine is built it detects those extra files and it too builds some extra feature (files) and installs those too. Fast forward into the and pretend that a hard dep for cups was added to minizip (crazy I know but it wants to print errors now). This time cups is built before re2 so abseil-cpp is not built so cups omits those files related to it being present. Now the footprint for qt6-webengine is corrupt. The extra files were not built by cups so qt6-webengine did not build its cups-related files either. Any depinst of qt6-webengine on a new CRUX will fail with MISSING files. This happened enough times when I used a VM and prt-get as my "clean" test system that I developed the docker method. In all cases leaving anything installed in the test environment will result in autoconfigure and friends detecting something and (not so) helpfully add extra files to their resulting tarballs possibly affecting other autoconfigure scripts downstream.
Note that although qt6-webengine lists numactl and pipewire as optional dependencies, and the test system did not have these ports installed, the prt-get command did not add them to the set of install targets. If you want a maximal port, you have to do more work to get it (see link at the end of this post).
Topological sorting is also modified in a sysup operation: my patched prt-get has the logic to update an optional dependency before the port that optionally depends on it (e.g., vulkan-loader before mesa, which is not the order you're guaranteed to get with the original prt-get). In fact, it was the optional dependencies of mesa that originally prompted my work on this fork.
I recall the discussion about optional dependencies. I was neither for or against it. Some might find it useful to automatically include them but I just review the ports and manually install an optionals I like.
I hope I've given a better explanation of what "softdeps yes" in the configuration file (or --softdeps on the command line) actually does. To reiterate: it does NOT automatically include as install targets all the ports in the "Optional:" field, it only applies a smarter sorting algorithm to the ports that are passed as arguments (and their hard dependencies). The expanded man-page demonstrates an unwieldy command if you want to build a "maximally interdependent" package, but this is not the default behavior when you set "softdeps yes".
Yes, much clearer now.
https://git.crux.nu/farkuhar/prt-get/src/branch/master/doc/prt-get.8#L719
-- John
With the docker environment I use there is a complete disconnect between the building of qt6-base and qt6-webengine. When qt6-base is built there is no qt6-webengine available. And when qt6-webengine is built only the qt6-base that was built independently is installed. So at build time of qt6-base there is no way to know that the optional dependency for cups in qt6-base is a hard dependency in qt6-webengine. Since 2017 this hasn't been a problem but in the meantime we've seen more conditional tests added to Pkgfiles that alter the configuration behaviour. In the case of qt6-webengine it is a good test because it clearly shows what is wrong before a lengthy compile that will end in a failure. But a hard dep in qt6-base with cups would have fixed that and my automation's problem too. When I think of all the bitcoin I could have mined with the electricity from failed Qt compiles :) -Daryl