Bugs in cat -n/-b and in bb_process_escape_sequence

[Sébastien Dusuel]

Hello,

I found two bugs, tested in busybox 1.27.2 running on ubuntu 18.04. The
bugs seem to also be in busybox 1.29.3 (from what could infer by reading
from the source).

* Bug 1 is a bug in `cat` when run with -n or -b option
To trigger this bug, simply try to cat a non-existent file, with -n or -b
option.
Although some error message is output, `cat` still tries to process the
file, resulting in a segfault (from what I can see in the source, because
it uses a FILE *fp which is NULL). For example:
$ cat -n A
cat: A: No such file or directory
Segmentation fault (core dumped)

* Bug 2 is a bug that affects many things because it occurs in the function
bb_process_escape_sequence defined in libbb/process_escape_sequence.c
Examples to trigger it:
$ echo -e "\x3@"
9
$ echo -e $(echo -e "\\x\x14\x11")
A

I wrote a little program using part of the code of the faulty function to
show where the problem occurs (you'll find it in the attachment). It shows
that characters \x10 to \x19, \x40, and \x60 are misintepreted as 0 to 9, 9
and 9 (respectively) when they are found in a \x escape sequence.

If it can help: decode_dollar_squote defined in shell/ash.c uses
bb_process_escape_sequence
but seems to be unaffected by the bug:
00000000: 7368 3a20 0340 3a20 6e6f 7420 666f 756e  sh: .@: not foun
00000010: 640a                                     d.


Best regards,
Sébastien Dusuel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/busybox/attachments/20181128/a8f414c5/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: bug.c
Type: text/x-csrc
Size: 481 bytes
Desc: not available
URL: <http://lists.busybox.net/pipermail/busybox/attachments/20181128/a8f414c5/attachment.c>