qt6-webengine: controlling ninja resource usage
Hi All! Based on the discussion on IRC and my own research I finally came up with a possible answer to memory depletion while compiling qt6-webengine. I added this to the Pkgfile: export NINJAFLAGS="-j $(( $(awk '/^MemAvailable:/ {print $2}' /proc/meminfo) / 3000000))" It determines the number of threads to use based on qt6-webengine using about 3GB per thread. It uses the memory available value rather than total memory to take into account memory used for desktop, etc. From the discussion on IRC it might be possible build without swapping with 2.5GB per thread but I haven't tested it. Interestingly not all invocations of ninja respect NINJAFLAGS. When building qt6-webengine with the above NINJAFLAGS I see one with -j8 (my machine has 8 threads with about 11GB available) and another with -j 4. I suspect it uses one ninja to parallel other ninja during the most memory intense part of the build there are only two ninja processes. I had been using a shim to compile qt6-webengine but it is cumbersome to use and affected all ninja builds so if I forgot to unshim I got slow builds. This solution might work for machines with widely varied resources but it will cause problems on very small machines. It can cause -j 0 in that case which will be bad since -j 0 means infinity. Probably not a good idea but I suspect a machine that small will not be used to build qt6-webengine anyway. Who has that that much time to wait? Because I use spinning disk for swap it is never a good idea to swap during builds. I've found that limiting the threads to never swap results in a faster compile (clock wise). I've read several complaints about ninja using nproc+2 for the default but the developers refuse to deal with it. I guess somewhere this an unwritten rule that REAL people have at least 4GB per thread. But I respect that, they probably have more than enough to do just to keep up with Qt without trying to get it to compile on every possible machine configuration. -Daryl
Hi! On 29.06.2024 21:04, Daryl F wrote:
Hi All!
Based on the discussion on IRC and my own research I finally came up with a possible answer to memory depletion while compiling qt6-webengine. I added this to the Pkgfile:
export NINJAFLAGS="-j $(( $(awk '/^MemAvailable:/ {print $2}' /proc/meminfo) / 3000000))"
It determines the number of threads to use based on qt6-webengine using about 3GB per thread. It uses the memory available value rather than total memory to take into account memory used for desktop, etc.
While something else I was trying created the problem in the first place for me, your proposed solution works fine for me! I'll add that to both qt5-webengine and qt6-webengine with the next push.
This solution might work for machines with widely varied resources but it will cause problems on very small machines. It can cause -j 0 in that case which will be bad since -j 0 means infinity. Probably not a good idea but I suspect a machine that small will not be used to build qt6-webengine anyway. Who has that that much time to wait?
I will probably just put a link to this mailing list thread into a README along the respective ports and I do agree, qt*-webengine is not for those in a hurry - but who am I kidding - a source based distro is probably not the right thing for that anyway.
I've read several complaints about ninja using nproc+2 for the default but the developers refuse to deal with it. I guess somewhere this an unwritten rule that REAL people have at least 4GB per thread. But I respect that, they probably have more than enough to do just to keep up with Qt without trying to get it to compile on every possible machine configuration.
I mean, under the hood, it's run by chromium, which is a killer to build and I guess a lot that's going on with qtwebengine builds stems from there, so one way to help improve the experience is to create tickets upstream and help to get them resolved.
-Daryl
Thanks for sharing Daryl and sorry for the long delay for this answer! Best regards, Tim
_______________________________________________ CRUX mailing list -- crux@lists.crux.nu To unsubscribe send an email to crux-leave@lists.crux.nu
participants (2)
-
Daryl F -
Tim Biermann