XZ embedded bug unpacking linux-3.8.tar.xz (was: Re: tar: short read on linux-3.8.tar.xz)

[Denys Vlasenko]

On Thu, Feb 28, 2013 at 1:34 AM, Matias A. Fonzo <selk at dragora.org> wrote:
> Hello Denys,
>
> El Wed, 27 Feb 2013 17:59:25 +0100
> Denys Vlasenko <vda.linux at googlemail.com> escribió:
>> On Mon, Feb 25, 2013 at 7:20 PM, Matias A. Fonzo <selk at dragora.org>
>> wrote:
>> > Can be lzip considered for inclusion in busybox?:
>> >
>> > [1] http://lzip.nongnu.org
>> > [2] http://en.wikipedia.org/wiki/Lzip
>> > [3]
>> > http://lists.busybox.net/pipermail/busybox/2012-December/078750.html
>> > [4] http://ur1.ca/810mp
>>
>> Matias, sure, this can be done.
>
> Great.
>
>> But bbox already has *two* LZMA decompressors.
>> Feels wrong, isn't it?
>>
>> In the long run it would be a nightmare to have two
>> or more LZMA (de)compressors in common use on Linux.
>
> Why?. *.lzma are deprecated some time ago

Because someone submitted the code:

commit c1d69906a0c5f28f3d84c14afb3b74c8f19f81c1
Author: Rob Landley <rob at landley.net>
Date:   Fri Jan 20 18:28:50 2006 +0000

    Patch from Aurelien Jacobs to add unlzma.  (A new decompression
type, see www.7-zip.org)


In fact, it is surprisingly small:

archival/libarchive:
   text    data     bss     dec     hex filename
   2827       0       0    2827     b0b decompress_unlzma.o
   7277       0       0    7277    1c6d decompress_unxz.o
   2743       0       0    2743     ab7 decompress_bunzip2.o
   5270       0       0    5270    1496 decompress_gunzip.o


>> What happened between lzip and xz? Are they incompatible?
>> On what level? File format, or compression stream format too?
>
> In both levels. In brief:
>
> Xz uses the method from LZMA SDK (By Igor Pavlov).
>
> Lzip uses a simplified version of the LZMA algorithm.
>
> About the file format, the first four bytes of a .lz file, says:
>
>   LZIP
>
> While the first four bytes of a .xz file contains a reference to
> "7zip". That is because Igor Pavlov (a Windows developer) was involved
> in the development of xz. To think in xz as an intent to introduced the
> Windows philosophy into Unix-like systems is: valid.

I don't really see advantages of one format over the other.


> Lzip was created before than xz. As a software distributor, I remember
> the existence of a project called "Tukaani" (a GNU/Linux distribution
> based on Slackware Linux featuring xz), in that time xz was in *beta
> state*. Meanwhile, Lzip's author was publishing and polishing stable
> versions of a LZMA implementation. Tukaani was in a lethargy, when
> Slackware decided to incorporate xz to distribute their packages; this
> example was followed by Fedora and other distributions, then Tukaani
> comes up to restart his activities (focusing only in the xz) -- After
> several months (if not years) of the xz creation, they announce the
> stable version...

So basically, lzip lost the race wrt adoption. xz is used more widely.
Kernel tarballs are .xz, not .lz.

What I'm saying is that bbox project would like to have is (ideally)
_one_ LZMA decoder. Unpacking the compressed stream from two formats
isn't a terribly difficult thing.

But can lzip decompress unxz *stream*?