Bugs in defconfig /bin/sh

[David N. Lombard]

On Sat, Oct 02, 2010 at 05:35:50PM -0700, Rob Landley wrote:
> On Thursday 30 September 2010 17:15:44 Denys Vlasenko wrote:
> > On Thursday 30 September 2010 23:29, Rob Landley wrote:
> > > > > (Making {file,file} curly bracket support work would be darn nice
> > > > > too, that's the biggest thing I miss.  You can't even follow the
> > > > > Linux From Scratch build instructions without that...)
> > > >
> > > > This is not easy. Consider three commands:
> > > >
> > > > v='/bin/*'; echo $v
> > > > v='/bin/{a,b}*'; echo $v
> > > > echo /bin/{a,b}*
> > > >
> > > > In the first case, unquoted $v is globbed *after* $v value is
> > > > substituted, and echo prints long list of filenames in /bin.
> > > >
> > > > In the second case, echo prints just "/bin/{a,b}*"
> > > >
> > > > In the third case, echo prints all filenames in /bin which start from a
> > > > and b.

According to the man page for bash 4.0.38:
  EXPANSION
    Expansion is performed on the command line after it has been split
    into words. There are seven kinds of expansion performed: brace
    expansion, tilde expansion, parameter and variable expansion,
    command substitution, arithmetic expansion, word splitting, and
    pathname expansion.

    The order of expansions is: brace expansion, tilde expansion,
    parameter, variable and arithmetic expansion and command
    substitution (done in a left-to-right fashion), word splitting,
    and pathname expansion.

    On systems that can support it, there is an additional expansion
    available: process substitution.

    Only brace expansion, word splitting, and pathname expansion can
    change the number of words of the expansion; other expansions
    expand a single word to a single word. The only exceptions to this
    are the expansions of "$@" and "${name[@]}" as explained above
    (see PARAMETERS).

Note variable expansion happens *after" brace expansion.  So, the
outputs you first described (expansion, literal brace/asterisk,
expansion) are correct.  Bash 2.05b, that didn't expand the braces and
filenames for the third case, was clearly broken.

This is consistent with Bourne and Korn, at least back to the early
'90's when I first started paying very close attention to the shell's
expanion behavior.

> > > That's messed up, and it really sounds like the second case is a bug.

No.

> > > I just fired up an Aboriginal system image with bash 2.05b (the much less
> > > bloated version), and under that the third case _also_ prints
> > > "/bin/{a,b}*", so the behavior of this corner case changed between bash
> > > versions.  Most likely somebody reported the third case as a bug and they
> > > fixed the one defect report without fixing the general case, because
> > > they're the FSF and everything is a special case to them.
> > >
> > > The _important_ test is just:
> > >
> > >   v='/bin/*'; echo $v
> > >
> > > Which answers the simple question of glob precedence: does it happen
> > > before or after the variable substitution?  And if the answer is "after"
> > > (which it is), then that's what we should do.  Perform globbing after
> > > substituting variables. {a,b} is part of globbing the same way * and ?
> > > and [x-y] and such are.

"{a,b}" is brace expansion; "*", "?", and "[x-y]" are pathname expansion.

[deletia]

> Wildcards can be inside the curly brackets as easily as outside:
> 
>   blah/{o?e,two.*}/*.txt

Yup, and it's easy to demonstrate:

  $ mkdir -p blah/ode blah/one blah/owe blah/two.three blah/two.z
  $ touch blah/ode/a.txt blah/one/a.txt \
  blah/owe/b.txt blah/two.three/a.txt blah/two.z/b.txt
  $ ls blah/{o?e,two.*}/*.txt
  blah/ode/a.txt  blah/owe/b.txt        blah/two.z/b.txt
  blah/one/a.txt  blah/two.three/a.txt
  $

> So either resolution happens at the same time, or you make multiple resolution 
> passes in which case keeping track of what was quoted and what wasn't (and 
> what was already expanded and what wasn't) is INSANE.

The latter *would* be insane.  Substitutions are done in the order:

  blah/{o?e,two.*}/*.txt
  blah/o?e/*.txt blah/two.*/*.txt
  blah/ode/a.txt blah/one/a.txt blah/owe/b.txt blah/two.three/a.txt blah/two.z/b.txt

[deletia]
> You're suggesting a need to be bug-for-bug compatible with something that 
> changes its behavior every version.  That's not a viable course of action.

I don't know how long the bug in 2.05b existed, but the current behavior
is consistent with the documented and traditional behavior, e.g.,
Bourne and Korn.

[deletia]
> I note that dash still doesn't handle {} at all, and thus can't be used to run 
> this bit of LFS:
> 
>   http://www.linuxfromscratch.org/lfs/view/6.7/chapter06/creatingdirs.html

dash is a problem all of its own...

-- 
David N. Lombard, Intel, Irvine, CA
I do not speak for Intel Corporation; all comments are strictly my own.