[rfc] slight optimization to prevent memory thrashing in file reading

Mike Frysinger vapier at gentoo.org
Thu Apr 12 00:26:56 UTC 2007


On Wednesday 11 April 2007, Denis Vlasenko wrote:
> You are referring to this code:
>
> char *bb_get_chunk_from_file(FILE * file, int *end)
> {
>         int ch;
>         int idx = 0;
>         char *linebuf = NULL;
>         int linebufsz = 0;
>
>         while ((ch = getc(file)) != EOF) {
>                 /* grow the line buffer as necessary */
>                 if (idx >= linebufsz) {
>                         linebuf = xrealloc(linebuf, linebufsz += 80);
>                 }
>                 linebuf[idx++] = (char) ch;
>                 if (!ch || (end && ch == '\n'))
>                         break;
>         }
>         if (end)
>                 *end = idx;
>         if (linebuf) {
>                 linebuf = xrealloc(linebuf, idx+1);
>                 linebuf[idx] = '\0';
>         }
>         return linebuf;
> }

yes

> Last realloc is used to truncate extra allocated data. Think about
> very long file with short lines (for example, empty lines) being read
> and stored in allocated memory using bb_get_chunk_from_file().

it looked to me like it was a fix for the 1 byte overflow for lines that were 
multiples of 80 bytes

> Without last realloc we may end up using lots more memory than really
> needed. Doesn't sound like good idea to me.

using grep as an example, the line is allocated, some work is done on it, and 
then it is freed before the next line is scanned in ... in this use case, 
memory is being thrashed by being pointless shrunk all of the time

> Maybe uclibc realloc() should be fixed instead?

ive already fixed it ... i wasnt suggesting this change because of that bug, 
that bug just lead me to this code snippet
-mike
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 827 bytes
Desc: not available
Url : http://lists.busybox.net/pipermail/busybox/attachments/20070411/038f4c0c/attachment-0002.pgp 


More information about the busybox mailing list