[bug] dd piped input truncates ~1% of values to %4096 on multi-core systems

[Nicholas Clark]

iflag=fullblock is the right solution. I added that feature a couple of
years ago. I thought it should be busybox's default behavior, but that goes
against the POSIX spec.

POSIX dd specifies that one block == one read() call, unless modified by an
iflag. Read() calls can return partial data pretty often, especially when
reading from pipes. A filesystem read() will _usually_ give you as much
data as you request, but it's not guaranteed at all.

So if you want to make _sure_ you're getting all of the data you want, the
only guaranteed safe approach is to use iflag=fullblock. It's good practice
to always use it with dd.



On Mon, Nov 30, 2020 at 2:11 AM John Thomson <
lists at johnthomson.fastmail.com.au> wrote:

> On Mon, 30 Nov 2020, at 08:39, Walter Harms wrote:
> > did you play with the buffering (man stdbuf) ?
> > I suspect the kernel has a more aggressive buffering these days.
> > In one project i had to add fflush() to make sure that i really get all
> > data. I never found a good explanation.
>
> I will look into it. Thank you.
>
> I have now found the fullblock iflag [1], which consistently gives me the
> expected result.
> Guessing the issue I was seeing relates to my bs being greater than
> PIPE_BUF?
>
> [1]:
> https://git.busybox.net/busybox/commit/coreutils/dd.c?id=77a6678c427db5ea15d6d0005a579f441277df06
>
> Cheers,
> --
>   John Thomson
> _______________________________________________
> busybox mailing list
> busybox at busybox.net
> http://lists.busybox.net/mailman/listinfo/busybox
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/busybox/attachments/20201130/a33ab26f/attachment-0001.html>