AW: [bug] dd piped input truncates ~1% of values to %4096 on multi-core systems
Walter Harms
wharms at bfs.de
Mon Nov 30 08:39:15 UTC 2020
Hi,
did you play with the buffering (man stdbuf) ?
I suspect the kernel has a more aggressive buffering these days.
In one project i had to add fflush() to make sure that i really get all
data. I never found a good explanation.
________________________________________
Von: busybox <busybox-bounces at busybox.net> im Auftrag von John Thomson <lists at johnthomson.fastmail.com.au>
Gesendet: Sonntag, 29. November 2020 14:21:00
An: busybox at busybox.net
Betreff: [bug] dd piped input truncates ~1% of values to %4096 on multi-core systems
Hi Busybox,
I may have run into a bug where dd with piped input will truncate a small number of results (~1%) to a modulo 4096 value on multi-core systems.
The test script does not produce errors running under bash.
This issue is not seen if dd reads the file with if=
I have not looked into the cause.
Example:
# generate test file
dd if=/dev/urandom of=/tmp/wlan_data_0 bs=1M count=1
# ~1% pipe error
cat /tmp/wlan_data_0 | \
dd iflag=skip_bytes bs=$((0x2f20)) skip=0 count=1 2>/dev/null | wc -c
First noticed on arm with busybox 1.31.1
Reproduced on x86_64 busybox g6a55b4e40
Reproduced on mipsel 4 core
Not reproduced on mips single core
cat data | dd used for Openwrt wifi radio calibration data extraction from sysfs:
https://github.com/openwrt/openwrt/commit/7557e7f267e845db5a403139c49f2637f9021992
Scripted example:
test_caldata_extract() {
local counter
local count_good
local count_bad
local result
local result_latest
local dd_bs
dd_bs=$((0x2f20))
dd if=/dev/urandom of=/tmp/wlan_data_0 bs=1M count=1
result=$(cat /tmp/wlan_data_0 | \
dd iflag=skip_bytes bs=$dd_bs skip=0 count=1 2>/dev/null | wc -c)
while [ ! $result -eq $dd_bs ]; do
echo "bad initial result"
result=$(cat /tmp/wlan_data_0 | \
dd iflag=skip_bytes bs=$dd_bs skip=0 count=1 2>/dev/null | wc -c)
done
counter=0
count_good=0
count_bad=0
echo "initial run $counter $result"
while true; do
counter=$(( counter + 1 ))
result_latest=$(cat /tmp/wlan_data_0 | \
dd iflag=skip_bytes bs=$((0x2f20)) skip=0 count=1 2>/dev/null | wc -c)
if [ "$result_latest" = "$result" ]; then
count_good=$(( count_good + 1 ))
#echo "run $counter okay"
else
count_bad=$(( count_bad + 1 ))
echo "run $counter bad $result_latest ($count_bad / $counter)"
#break
fi
done
}
test_caldata_extract
Cheers
--
John Thomson
_______________________________________________
busybox mailing list
busybox at busybox.net
http://lists.busybox.net/mailman/listinfo/busybox
More information about the busybox
mailing list