Let tar extract files > 8G

[Denys Vlasenko]

Hi Ian!


On Thursday 11 December 2008 22:04, Ian Wienand wrote:
> The 12 octal digits for size give each file a maximum size 8G.  GNU
> tar encodes sizes larger than this with a base-256 encoding, and marks
> the top bit of the size set to indicate this.

I thought tar uses pax extensions for this...
but apparently not.

> I cribbed the base 256 converter from gnu tar sources

It shows that GNU sources were used as a base. ~180 bytes of code.

> +/* Extract a number encoded in base-256 */
> +static unsigned long long getBase256(char *str, int len)
> +{
> +#define LG_256 8

Non-8-bit-char machines are history.

> +       uintmax_t value;

1. Why unitmax_t if you plan to return unsigned long long anyway?
   ...which then will become off_t... 

2. Indentation by seven spaces is a novelty.

> +       char *lim = str + len;
> +       /* Parse base-256 output.  A nonnegative number N is
> +          represented as (256**DIGS)/2 + N; a negative number -N is
> +          represented as (256**DIGS) - N, i.e. as two's complement.

1. I didn't understand much.

2. File sizes can be negative?! wow.

> +          The representation guarantees that the leading bit is
> +          always on, so that we don't confuse this format with the
> +          others (assuming ASCII bytes of 8 bits or more).  */
> +       int signbit = *str & (1 << (LG_256 - 2));

How about (*str & 0x40) or (*str & (1 << 6)) for readability?

> +       uintmax_t topbits = (((uintmax_t) - signbit)
> +                            << (CHAR_BIT * sizeof (uintmax_t)
> +                                - LG_256 - (LG_256 - 2)));

Timing how fast I will grok it...

(0xffff...fffffc0 or 0) << (all bits - 14)
so,
0xff0000...00000 or 0

Took 5 minutes!

Let's look further. If signbit != 0, it's ALREADY something
bad (negative filesize!!!), if not, topbits is 0, why calculate it?

> +       value = (*str++ & ((1 << (LG_256 - 2)) - 1)) - signbit;
> +       for (;;)
> +       {
> +               value = (value << LG_256) + (unsigned char) *str++;
> +               if (str == lim)
> +                       break;
> +               if (((value << LG_256 >> LG_256) | topbits) != value)

Especially sadistic method to check for overflows.

...
> -	file_header->size = GET_OCTAL(tar.size);
> +
> +	/* Really big (> 8 gig) files have the size encoded in base
> +	 * 256.  The top bit being set tells us this ... */
> +	file_header->size = (*tar.size == '\200') ?

*tar.size == '\200'  is not  "top bit set".


Ok. Enough of pointless ranting for today :)

I reworked it so that it compiles into ~15 instructions on x86.
Also fixed a bug where we did not print lenghts correctly
in "tar tvf" output (were truncating them to unsigned ints),
tested and committed it to svn.

Thanks!
--
vda