Hi, CRUX 2.6 will ship with XZ (http://tukaani.org/xz/) in core. That package implements the XZ compression algorithm, which is the successor to LZMA. I believe that LZMA isn't in use in the free software world anymore, so I will only talk about XZ here. To extract an xz'ed tarball, you can do as follows: $ tar xJf coreutils-7.4.tar.xz This is obviously independent of libarchive ;) To extract an xz'ed tarball using bsdtar (libarchive), there's two choices basically: a) tell bsdtar to invoke unxz on the file to uncompress it, then just extract the uncompressed tarball: $ bsdtar --use-compress-program unxz xf coreutils-7.4.tar.xz b) do both uncompression and extraction using libarchive. $ bsdtar xf coreutils-7.4.tar.xz You can optionally specify the J flag to force XZ compression. This requires that libarchive was built with XZ support. Since xz is in core anyway, I would like to build our libarchive with XZ support. Your opinion? What does that mean for pkgutils? At the moment, in pkgutils we initialize libarchive so that support for all supported archive formats and compression algorithms gets included. This means that if compression algorithm foo is implemented in libfoo and libarchive uses libfoo to handle foo we need to link pkgutils statically to libfoo. Until now, this only meant we needed to pull in libz and libbz2. Now, if your copy of libarchive is built with XZ support it will be linked to liblzma (liblzma implements the XZ compression format. Yes, the name is confusing). So if you initialize libarchive to support all known archive/compression formats but only link to libz and libbz2 you'll get a linker error because the XZ stuff isn't found. There's two possible solutions for this: a) link to liblzma (XZ!) too. This will increase the size of our pkgadd binary since now a few more object files from liblzma.a will be included. b) initialize libarchive so that support for XZ is disabled -> no need to link to liblzma. This means that pkgadd/pkginfo will not be able to open xz'ed tarballs. Note that this does NOT apply to pkgmk. pkgmk doesn't use libarchive obviously, but instead calls out to tar to extract stuff. In case we decide to build bsdtar with libarchive support, I would like to change pkgmk so that it uses bsdtar to extract *.tar.gz, *.tar.Z, *.tgz, *.tar.bz2, *.tar.xz, *.zip Let me know what you think. Regards, Tilman -- A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing? A: Top-posting. Q: What is the most annoying thing on usenet and in e-mail?
Tilman Sauerbeck [2009-06-14 13:23]:
a) link to liblzma (XZ!) too. This will increase the size of our pkgadd binary since now a few more object files from liblzma.a will be included.
Some numbers: $ size pkgadd-without-xz text data bss dec hex filename 1369326 7884 38224 1415434 15990a pkgadd-without-xz $ size pkgadd-with-xz text data bss dec hex filename 1438142 8096 38224 1484462 16a6ae pkgadd-with-xz $ ls -l pkgadd-without-xz pkgadd-with-xz -rwxr-xr-x 1 root root 1.4M May 19 02:45 pkgadd-without-xz -rwxr-xr-x 1 tilman tilman 1.8M Jun 14 13:27 pkgadd-with-xz Regards, Tilman -- A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing? A: Top-posting. Q: What is the most annoying thing on usenet and in e-mail?
On Sun, Jun 14, 2009 at 01:23:37PM +0200, Tilman Sauerbeck wrote: Hi Tilman, [...]
There's two possible solutions for this: a) link to liblzma (XZ!) too. This will increase the size of our pkgadd binary since now a few more object files from liblzma.a will be included. b) initialize libarchive so that support for XZ is disabled -> no need to link to liblzma. This means that pkgadd/pkginfo will not be able to open xz'ed tarballs.
I don't think that we need this. Changing our package-format to xz might be an nice option, but looking at typical compression times is really disappointing. Here's a quick test with our latest xulrunner package: $:> mkdir foo $:> tar -C foo -xf xulrunner#1.9.0.11-1.pkg.tar.gz $:> time tar -czf foo.tar.gz foo real 0m4.409s user 0m4.250s sys 0m0.150s $:> time tar -cJf foo.tar.xz foo # -9 compression real 1m11.419s user 1m10.950s sys 0m0.570s $:> echo 'scale=2; (60+11.42)/4.41' | bc 16.19 $:> ls -l *.gz *.xz -rw-r--r-- 1 juergen users 18224116 Jun 15 10:02 foo.tar.gz -rw-r--r-- 1 juergen users 12979240 Jun 15 10:04 foo.tar.xz
Note that this does NOT apply to pkgmk. pkgmk doesn't use libarchive obviously, but instead calls out to tar to extract stuff. In case we decide to build bsdtar with libarchive support, I would like to change pkgmk so that it uses bsdtar to extract *.tar.gz, *.tar.Z, *.tgz, *.tar.bz2, *.tar.xz, *.zip
Let me know what you think.
Having bz2 and xz support in libarchive makes only sense IMO if other programs like bsdtar can use that support, obviously we don't need it for pkgadd. On the other side libarchive is a general purpose library and should support all compression programs we have in core. We should not care too much about the increase in filesize of pkgadd. For me most important is the support of *.xz/*.lzma source tarballs, distfiles, in pkgmk. To summarize I would link libarchive against liblzma and have no objections against using bsdtar in pkgmk. Greetings Juergen -- Juergen Daubert | mailto:jue@jue.li Korb, Germany | http://jue.li/crux
On Mon, Jun 15, 2009 at 10:39 AM, Juergen Daubert<jue@jue.li> wrote:
On Sun, Jun 14, 2009 at 01:23:37PM +0200, Tilman Sauerbeck wrote:
Hi Tilman,
[...]
There's two possible solutions for this: a) link to liblzma (XZ!) too. This will increase the size of our pkgadd binary since now a few more object files from liblzma.a will be included. b) initialize libarchive so that support for XZ is disabled -> no need to link to liblzma. This means that pkgadd/pkginfo will not be able to open xz'ed tarballs.
I don't think that we need this. Changing our package-format to xz might be an nice option, but looking at typical compression times is really disappointing.
Here's a quick test with our latest xulrunner package:
$:> mkdir foo $:> tar -C foo -xf xulrunner#1.9.0.11-1.pkg.tar.gz
$:> time tar -czf foo.tar.gz foo real 0m4.409s user 0m4.250s sys 0m0.150s
$:> time tar -cJf foo.tar.xz foo # -9 compression real 1m11.419s user 1m10.950s sys 0m0.570s
$:> echo 'scale=2; (60+11.42)/4.41' | bc 16.19
$:> ls -l *.gz *.xz -rw-r--r-- 1 juergen users 18224116 Jun 15 10:02 foo.tar.gz -rw-r--r-- 1 juergen users 12979240 Jun 15 10:04 foo.tar.xz
Note that this does NOT apply to pkgmk. pkgmk doesn't use libarchive obviously, but instead calls out to tar to extract stuff. In case we decide to build bsdtar with libarchive support, I would like to change pkgmk so that it uses bsdtar to extract *.tar.gz, *.tar.Z, *.tgz, *.tar.bz2, *.tar.xz, *.zip
Let me know what you think.
Having bz2 and xz support in libarchive makes only sense IMO if other programs like bsdtar can use that support, obviously we don't need it for pkgadd.
It's a one line change to make pkgadd only initialize gzip decompression, avoiding statically linking in all of libarchive's supported formats. This would keep libarchive nice and generic while not bloating pkgadd with every decompression library known to mankind. I'm not sure .pkg.bz2 packages even work though pkgadd already pulls in bz2. Last time I checked (admittedly long ago, around the libtar -> libarchive port) the package extension was hardcoded.
Mark Rosenstand [2009-06-16 13:19]:
On Mon, Jun 15, 2009 at 10:39 AM, Juergen Daubert<jue@jue.li> wrote:
On Sun, Jun 14, 2009 at 01:23:37PM +0200, Tilman Sauerbeck wrote:
Hi Tilman,
[...]
There's two possible solutions for this: a) link to liblzma (XZ!) too. This will increase the size of our pkgadd binary since now a few more object files from liblzma.a will be included. b) initialize libarchive so that support for XZ is disabled -> no need to link to liblzma. This means that pkgadd/pkginfo will not be able to open xz'ed tarballs.
I don't think that we need this. Changing our package-format to xz might be an nice option, but looking at typical compression times is really disappointing.
Here's a quick test with our latest xulrunner package:
$:> mkdir foo $:> tar -C foo -xf xulrunner#1.9.0.11-1.pkg.tar.gz
$:> time tar -czf foo.tar.gz foo real 0m4.409s user 0m4.250s sys 0m0.150s
$:> time tar -cJf foo.tar.xz foo # -9 compression real 1m11.419s user 1m10.950s sys 0m0.570s
$:> echo 'scale=2; (60+11.42)/4.41' | bc 16.19
$:> ls -l *.gz *.xz -rw-r--r-- 1 juergen users 18224116 Jun 15 10:02 foo.tar.gz -rw-r--r-- 1 juergen users 12979240 Jun 15 10:04 foo.tar.xz
Note that this does NOT apply to pkgmk. pkgmk doesn't use libarchive obviously, but instead calls out to tar to extract stuff. In case we decide to build bsdtar with libarchive support, I would like to change pkgmk so that it uses bsdtar to extract *.tar.gz, *.tar.Z, *.tgz, *.tar.bz2, *.tar.xz, *.zip
Let me know what you think.
Having bz2 and xz support in libarchive makes only sense IMO if other programs like bsdtar can use that support, obviously we don't need it for pkgadd.
It's a one line change to make pkgadd only initialize gzip decompression, avoiding statically linking in all of libarchive's supported formats.
We know (and I already did the change before posting this).
I'm not sure .pkg.bz2 packages even work though pkgadd already pulls in bz2. Last time I checked (admittedly long ago, around the libtar -> libarchive port) the package extension was hardcoded.
It still is. Regards, Tilman -- A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing? A: Top-posting. Q: What is the most annoying thing on usenet and in e-mail?
participants (3)
-
Juergen Daubert -
Mark Rosenstand -
Tilman Sauerbeck