[bug] dd piped input truncates ~1% of values to %4096 on multi-core systems

[Denys Vlasenko]

On Sun, Nov 29, 2020 at 2:21 PM John Thomson
<lists at johnthomson.fastmail.com.au> wrote:
> I may have run into a bug where dd with piped input will truncate a small number of results (~1%) to a modulo 4096 value on multi-core systems.
> The test script does not produce errors running under bash.
> This issue is not seen if dd reads the file with if=
> I have not looked into the cause.
>
> Example:
> # generate test file
> dd if=/dev/urandom of=/tmp/wlan_data_0 bs=1M count=1
> # ~1% pipe error
> cat /tmp/wlan_data_0 | \
> dd iflag=skip_bytes bs=$((0x2f20)) skip=0 count=1 2>/dev/null | wc -c

Pipes do not guarantee to buffer anything more than PIPE_BUF bytes
(usually 4096). Your bs is larger than that.

On multi-core machine, both "cat" and "dd" run simultaneously.
If cat writes less than $bs number of bytes in one go, or if kernel buffers
less than $bs number of bytes (even if cat tries to write more than that
at once in one write() call), then "dd" can see a nonzero, but less than $bs
number of bytes and consume them - its read() call will see a short read.