AW: [bug] dd piped input truncates ~1% of values to %4096 on multi-core systems

Walter Harms wharms at bfs.de
Mon Nov 30 08:39:15 UTC 2020


Hi,
did you play with the buffering (man stdbuf) ?

I suspect the kernel has a more aggressive buffering these days.
In one project i had to add fflush() to make sure that i really get all
data. I never found a good explanation.
________________________________________
Von: busybox <busybox-bounces at busybox.net> im Auftrag von John Thomson <lists at johnthomson.fastmail.com.au>
Gesendet: Sonntag, 29. November 2020 14:21:00
An: busybox at busybox.net
Betreff: [bug] dd piped input truncates ~1% of values to %4096 on multi-core systems

Hi Busybox,

I may have run into a bug where dd with piped input will truncate a small number of results (~1%) to a modulo 4096 value on multi-core systems.
The test script does not produce errors running under bash.
This issue is not seen if dd reads the file with if=
I have not looked into the cause.

Example:
# generate test file
dd if=/dev/urandom of=/tmp/wlan_data_0 bs=1M count=1
# ~1% pipe error
cat /tmp/wlan_data_0 | \
dd iflag=skip_bytes bs=$((0x2f20)) skip=0 count=1 2>/dev/null | wc -c

First noticed on arm with busybox 1.31.1
Reproduced on x86_64 busybox g6a55b4e40
Reproduced on mipsel 4 core
Not reproduced on mips single core

cat data | dd used for Openwrt wifi radio calibration data extraction from sysfs:
https://github.com/openwrt/openwrt/commit/7557e7f267e845db5a403139c49f2637f9021992

Scripted example:
test_caldata_extract() {
        local counter
        local count_good
        local count_bad
        local result
        local result_latest
        local dd_bs

        dd_bs=$((0x2f20))

        dd if=/dev/urandom of=/tmp/wlan_data_0 bs=1M count=1
        result=$(cat /tmp/wlan_data_0 | \
        dd iflag=skip_bytes bs=$dd_bs skip=0 count=1 2>/dev/null | wc -c)
        while [ ! $result -eq $dd_bs ]; do
        echo "bad initial result"
        result=$(cat /tmp/wlan_data_0 | \
        dd iflag=skip_bytes bs=$dd_bs skip=0 count=1 2>/dev/null | wc -c)
        done

        counter=0
        count_good=0
        count_bad=0
        echo "initial run $counter $result"

        while true; do
        counter=$(( counter + 1 ))
        result_latest=$(cat /tmp/wlan_data_0 | \
        dd iflag=skip_bytes bs=$((0x2f20)) skip=0 count=1 2>/dev/null | wc -c)
        if [ "$result_latest" = "$result" ]; then
        count_good=$(( count_good + 1 ))
        #echo "run $counter okay"
        else
        count_bad=$(( count_bad + 1 ))
        echo "run $counter bad $result_latest ($count_bad / $counter)"
        #break
        fi
        done
}

test_caldata_extract


Cheers
--
  John Thomson
_______________________________________________
busybox mailing list
busybox at busybox.net
http://lists.busybox.net/mailman/listinfo/busybox


More information about the busybox mailing list