Precompiled headers and distributed compilation
Hi there, My qt4 port triggered an error [1] when using distcc to build it. After some investigation, I realized that Qt4 by default uses precompiled headers [2], which break when the compilation is done on a different machine since the precompiled headers are not transfered automatically. Currently, this means that any package using PCH will break when trying to compile it with distcc. There are a couple of ways to address that: 1. let it break on distcc machines, since we expect users of distcc to understand the problem 2. force the packages to not use PCH at all (= longer compile times for non distributed builds) 3. make the Pkgfiles aware that they're build with distcc, so they can react (might need a more generic solution than just distcc, since other tools doing the same would suffer from the same problem) 4. make the port aware (i.e. using a ".pch" file) that it will break, and make the distcc code in pkgmk.conf smart enough to not select distcc if it's there I think #4 would be the cleanest solution, however #3 would be faster if a user has a compile farm, at the cost of some detection code build() function At this point in time, distcc is not maintained in core/opt, so I'm not sure if there's a general interest to fix that problem; I used to maintain distcc before my "sabbatical leave" so I have a personal interest in it and believe that distcc is quite interesting for a source based distribution, YMMV though :-). If you have other ideas or any preferences, please tell the world :-). Best wishes, Johannes References: 1. http://crux.nu/bugs/index.php?do=details&task_id=275 2. http://gcc.gnu.org/onlinedocs/gcc/Precompiled-Headers.html -- Johannes Winkelmann mailto:jw@smts.ch Zurich, Switzerland http://jw.smts.ch
Johannes Winkelmann wrote:
Hi there,
My qt4 port triggered an error [1] when using distcc to build it. After some investigation, I realized that Qt4 by default uses precompiled headers [2], which break when the compilation is done on a different machine since the precompiled headers are not transfered automatically.
Currently, this means that any package using PCH will break when trying to compile it with distcc.
There are a couple of ways to address that:
1. let it break on distcc machines, since we expect users of distcc to understand the problem 2. force the packages to not use PCH at all (= longer compile times for non distributed builds) 3. make the Pkgfiles aware that they're build with distcc, so they can react (might need a more generic solution than just distcc, since other tools doing the same would suffer from the same problem) 4. make the port aware (i.e. using a ".pch" file) that it will break, and make the distcc code in pkgmk.conf smart enough to not select distcc if it's there
I think #4 would be the cleanest solution, however #3 would be faster if a user has a compile farm, at the cost of some detection code build() function
I like option 3, it's not that hard to detect although distcc and ccach functionality should be apart of pkgutils.
At this point in time, distcc is not maintained in core/opt, so I'm not sure if there's a general interest to fix that problem; I used to maintain distcc before my "sabbatical leave" so I have a personal interest in it and believe that distcc is quite interesting for a source based distribution, YMMV though :-).
Just because Tilman took over distcc then later dropped it and I was using it so I picked it up and put it in contrib. Then you return and I'm sure you would grab distcc at the first chance you got. I would consider puting a number of my ports in opt that i thnk beling there but Tilman wont allow me in opt so thats not going to happen. I'm forced to give away my ports, and you saying in IRC that most would be happy to give a port away. Yes and no depends how much I use it, Clustering is an area that interests me. I would guard my ports but it seems I'll lose some ports like mpfr later since gcc will require it in later versions.
If you have other ideas or any preferences, please tell the world :-). Best wishes, Johannes
References: 1. http://crux.nu/bugs/index.php?do=details&task_id=275 2. http://gcc.gnu.org/onlinedocs/gcc/Precompiled-Headers.html
Regards, Danny Rawlins <Romster>
Danny Rawlins [2008-05-19 13:42]:
area that interests me. I would guard my ports but it seems I'll lose some ports like mpfr later since gcc will require it in later versions.
That seems to be a huge problem for you, which I really don't understand. What's the deal about not maintaining one of a couple of hundred ports anymore? It's not like it will be dumped, it will continue to be maintained in core. Regards, Tilman -- A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing? A: Top-posting. Q: What is the most annoying thing on usenet and in e-mail?
Hi, On Mon, May 19, 2008 at 13:42:18 +1000, Danny Rawlins wrote:
Johannes Winkelmann wrote:
Hi there, [...] There are a couple of ways to address that:
3. make the Pkgfiles aware that they're build with distcc, so they can react (might need a more generic solution than just distcc, since other tools doing the same would suffer from the same problem)
I like option 3, it's not that hard to detect Well, currently build() should be independent from the set of installed packages, so it's not so much about being "hard" but being a change of packaging practices, and finding an elegant way to do so.
I know opt/qt3 currently breaks that rule by calling pkginfo, I hope we can fix that in the future.
although distcc and ccach functionality should be apart of pkgutils. Yeah, I agree that it should remain outside of pkgutils, the current way is a good example of KISS concept: minimal features by default, but extensible when wanted.
At this point in time, distcc is not maintained in core/opt, so I'm not sure if there's a general interest to fix that problem; I used to maintain distcc before my "sabbatical leave" so I have a personal interest in it and believe that distcc is quite interesting for a source based distribution, YMMV though :-).
Just because Tilman took over distcc then later dropped it and I was using it so I picked it up and put it in contrib. Then you return and I'm sure you would grab distcc at the first chance you got. I certainly won't pick it up for the sake of maintaining it, and have no plans to do it at this point in time.
To me, it's really about having the software available (in proper quality). Maintaining it is just the way, not the goal.
I would consider puting a number of my ports in opt that i thnk beling there but Tilman wont allow me in opt so thats not going to happen. Well, Tilman alone does not make the call who gets commit access and who doesn't (he's part of the group that does that, though).
I'm forced to give away my ports and you saying in IRC that most would be happy to give a port away. Yes and no depends how much I use it, Clustering is an area that interests me. Even if we'd promote some port to core/opt, how does take away the port from you? If maintaining the port is vital to you, you can keep doing
Also note that _maintainers_ (i.e. persons) are selected to maintain ports in opt, not the ports themselves. Check [1], "Stage 3" for more information on that. that in your private repository. In the end you should be happy since it's apparently important to you that _you_ maintain it, and the users are happy since the port itself is still as easily available as before (or maybe easier) and maybe even offering an improved overall experience e.g. if more ports get tested with distcc. A typical win-win situation. And once your interest moves from "groom a port for distributed compilation" to "distributed compilation" itself, you can move to the opt port and use the saved time to study the application instead of the packaging. Regards, Johannes Reference: 1. http://crux.nu/Main/HowToContribute -- Johannes Winkelmann mailto:jw@smts.ch Zurich, Switzerland http://jw.smts.ch
Johannes Winkelmann wrote:
Hi,
On Mon, May 19, 2008 at 13:42:18 +1000, Danny Rawlins wrote:
Johannes Winkelmann wrote:
Hi there,
[...]
There are a couple of ways to address that:
3. make the Pkgfiles aware that they're build with distcc, so they can react (might need a more generic solution than just distcc, since other tools doing the same would suffer from the same problem)
I like option 3, it's not that hard to detect
Well, currently build() should be independent from the set of installed packages, OK but... configure scripts can also detect if a program is installed or not. Does this also imply that we should use --disable-foo and --without-foo on other optional packages not included in the 'depends on' line or pulled in from the dependency tree? I don't see a difference here. so it's not so much about being "hard" but being a change of packaging practices, and finding an elegant way to do so.
I know opt/qt3 currently breaks that rule by calling pkginfo, I hope we can fix that in the future.
although distcc and ccach functionality should be apart of pkgutils.
Yeah, I agree that it should remain outside of pkgutils, the current way is a good example of KISS concept: minimal features by default, but extensible when wanted.
At this point in time, distcc is not maintained in core/opt, so I'm not sure if there's a general interest to fix that problem; I used to maintain distcc before my "sabbatical leave" so I have a personal interest in it and believe that distcc is quite interesting for a source based distribution, YMMV though :-).
Just because Tilman took over distcc then later dropped it and I was using it so I picked it up and put it in contrib. Then you return and I'm sure you would grab distcc at the first chance you got.
I certainly won't pick it up for the sake of maintaining it, and have no plans to do it at this point in time.
To me, it's really about having the software available (in proper quality). Maintaining it is just the way, not the goal.
Yes and when I maintain I know what to expect. But I can also keep my own copy of a port, which i do in say cases of imagemagick being a old version and where I apply my own personal patches like with my xchat.
I would consider puting a number of my ports in opt that i thnk beling there but Tilman wont allow me in opt so thats not going to happen.
Well, Tilman alone does not make the call who gets commit access and who doesn't (he's part of the group that does that, though).
Also note that _maintainers_ (i.e. persons) are selected to maintain ports in opt, not the ports themselves. Check [1], "Stage 3" for more information on that.
I'm forced to give away my ports and you saying in IRC that most would be happy to give a port away. Yes and no depends how much I use it, Clustering is an area that interests me.
Even if we'd promote some port to core/opt, how does take away the port from you? It takes away how I would keep it, and if some change did occur that I disliked I would have to copy and maintain my own copy when I could of just kept the port in question in the first place. If maintaining the port is vital to you, you can keep doing that in your private repository. In the end you should be happy since it's apparently important to you that _you_ maintain it, and the users are happy since the port itself is still as easily available as before (or maybe easier) and maybe even offering an improved overall experience e.g. if more ports get tested with distcc. A typical win-win situation.
Tested is good and I see the good side of it too.
And once your interest moves from "groom a port for distributed compilation" to "distributed compilation" itself, you can move to the opt port and use the saved time to study the application instead of the packaging.
Regards, Johannes
Reference: 1. http://crux.nu/Main/HowToContribute
On Tue, May 20, 2008 at 22:27:30 +1000, Danny Rawlins wrote:
Johannes Winkelmann wrote: [...]
Well, currently build() should be independent from the set of installed packages, OK but... configure scripts can also detect if a program is installed or not. Does this also imply that we should use --disable-foo and --without-foo on other optional packages not included in the 'depends on' line or pulled in from the dependency tree? I don't see a difference here.
If you compare the following two Pkgfiles, I'm sure you'll see the difference: http://www.xs4all.nl/~hanb/software/crux/han/transcode/Pkgfile http://crux.nu/gitweb/?p=ports/contrib.git;a=blob_plain;f=transcode/Pkgfile;... HTH, Johannes -- Johannes Winkelmann mailto:jw@smts.ch Zurich, Switzerland http://jw.smts.ch
On Sun, May 18, 2008 at 12:32:09 +0200, Johannes Winkelmann wrote:
Hi there,
My qt4 port triggered an error [1] when using distcc to build it. After some investigation, I realized that Qt4 by default uses precompiled headers [2], which break when the compilation is done on a different machine since the precompiled headers are not transfered automatically.
Currently, this means that any package using PCH will break when trying to compile it with distcc.
As a heads up, there's a new maintainer for distcc, and apparently one of the features to come soonish is support for precompiled headers. More info here: http://sourcefrog.net/weblog/2008/06/19/#passing-baton Good news for the distcc crowd :-). Regards, Johannes -- Johannes Winkelmann mailto:jw@smts.ch Zurich, Switzerland http://jw.smts.ch
participants (3)
-
Danny Rawlins -
Johannes Winkelmann -
Tilman Sauerbeck