inotifyd - possible memory leak?

Sat Nov 28 18:30:21 UTC 2009

Please do not remove busybox at busybox.net from CC.

On Saturday 28 November 2009 17:39, Piotr Grudzinski wrote:
> > msh does not avoid fragmentation, it just doesn't produce as much of it.
> >
> > I would happily take patches which either fix msh (it is VERY buggy)
> > or make hush better wrt fragmentation.
> 
> Just in case I have some extra time to spare, any pointers to:
> - what is wrong with msh?

Build busybox with hush enabled with all bells and whistles.
cd shell/hush_test and run ./run-all. You should get this output:

# ./run-all
No ./hush - creating a link to ../../busybox
hush-arith/arith.tests: ok
hush-bugs/and_or_and_backgrounding.tests: fail
hush-bugs/export_exp.tests: fail
hush-glob/glob1.tests: ok
hush-glob/glob_and_assign.tests: ok
hush-glob/glob_redir.tests: ok
hush-leak/leak_argv1.tests: ok
hush-misc/and-or.tests: ok
hush-misc/assignment1.tests: ok
hush-misc/break1.tests: ok
hush-misc/break2.tests: ok
hush-misc/break3.tests: ok
hush-misc/break4.tests: ok
hush-misc/break5.tests: ok
hush-misc/builtin1.tests: ok
hush-misc/case1.tests: ok
hush-misc/colon.tests: ok
hush-misc/compound.tests: ok
hush-misc/continue1.tests: ok
hush-misc/continue2.tests: ok
hush-misc/continue3.tests: ok
hush-misc/empty_args.tests: ok
hush-misc/empty_for.tests: ok
hush-misc/empty_for2.tests: ok
hush-misc/env_and_func.tests: ok
hush-misc/exec.tests: ok
hush-misc/exit1.tests: ok
hush-misc/export-n.tests: ok
hush-misc/export.tests: ok
hush-misc/for_with_bslashes.tests: ok
hush-misc/for_with_keywords.tests: ok
hush-misc/func1.tests: ok
hush-misc/func2.tests: ok
hush-misc/func3.tests: ok
hush-misc/func4.tests: ok
hush-misc/func5.tests: ok
hush-misc/func_args1.tests: ok
hush-misc/func_local1.tests: ok
hush-misc/func_local2.tests: ok
hush-misc/heredoc1.tests: ok
hush-misc/heredoc2.tests: ok
hush-misc/heredoc3.tests: ok
hush-misc/heredoc_huge.tests: ok
hush-misc/if_false_exitcode.tests: ok
hush-misc/nommu1.tests: ok
hush-misc/opts1.tests: ok
hush-misc/pid.tests: ok
hush-misc/read.tests: ok
hush-misc/redir1.tests: ok
hush-misc/redir2.tests: ok
hush-misc/redir3.tests: ok
hush-misc/redir4.tests: ok
hush-misc/redir5.tests: ok
hush-misc/redir6.tests: ok
hush-misc/shift.tests: ok
hush-misc/syntax_err.tests: ok
hush-misc/syntax_err_negate.tests: ok
hush-misc/until1.tests: ok
hush-misc/while1.tests: ok
hush-misc/while2.tests: ok
hush-misc/while_in_subshell.tests: ok
hush-parsing/argv0.tests: ok
hush-parsing/brace1.tests: ok
hush-parsing/brace2.tests: ok
hush-parsing/escape1.tests: ok
hush-parsing/escape2.tests: ok
hush-parsing/escape3.tests: ok
hush-parsing/escape4.tests: ok
hush-parsing/escape5.tests: ok
hush-parsing/group1.tests: ok
hush-parsing/groups_and_keywords1.tests: ok
hush-parsing/negate.tests: ok
hush-parsing/noeol.tests: ok
hush-parsing/noeol2.tests: ok
hush-parsing/noeol3.tests: ok
hush-parsing/process_subst.tests: ok
hush-parsing/quote1.tests: ok
hush-parsing/quote2.tests: ok
hush-parsing/quote3.tests: ok
hush-parsing/quote4.tests: ok
hush-parsing/redir_space.tests: ok
hush-parsing/starquoted.tests: ok
hush-parsing/starquoted2.tests: ok
hush-psubst/emptytick.tests: ok
hush-psubst/tick.tests: ok
hush-psubst/tick2.tests: ok
hush-psubst/tick3.tests: ok
hush-psubst/tick4.tests: ok
hush-psubst/tick5.tests: ok
hush-psubst/tick_huge.tests: ok
hush-trap/catch.tests: ok
hush-trap/exit.tests: ok
hush-trap/save-ret.tests: ok
hush-trap/savetrap.tests: ok
hush-trap/subshell.tests: ok
hush-trap/usage.tests: ok
hush-vars/empty.tests: ok
hush-vars/glob_and_vars.tests: ok
hush-vars/param_expand_alt.tests: ok
hush-vars/param_expand_assign.tests: ok
hush-vars/param_expand_default.tests: ok
hush-vars/param_expand_indicate_error.tests: ok
hush-vars/param_expand_len.tests: ok
hush-vars/param_glob.tests: ok
hush-vars/param_subshell.tests: ok
hush-vars/star.tests: ok
hush-vars/unset.tests: ok
hush-vars/var1.tests: ok
hush-vars/var2.tests: ok
hush-vars/var3.tests: ok
hush-vars/var_expand_in_assign.tests: ok
hush-vars/var_expand_in_redir.tests: ok
hush-vars/var_in_pipes.tests: ok
hush-vars/var_leaks.tests: ok
hush-vars/var_posix1.tests: ok
hush-vars/var_preserved.tests: ok
hush-vars/var_subst_in_for.tests: ok
hush-z_slow/leak_all1.tests: ok
hush-z_slow/leak_all2.tests: ok
hush-z_slow/leak_heredoc1.tests: ok
hush-z_slow/leak_var.tests: ok
hush-z_slow/leak_var2.tests: ok
hush-z_slow/leak_var3.tests: ok

Now, rename msh_unused.c to hush.c, and in it rename msh_main()
to hush_main(). Rebuild busybox. Rerun ./run-all.

I just tried it and I've got 57 failures before msh entered
an infinite loop in one of the tests (I think it's
hush-psubst/tick_huge.tests).

> - what to look for in hush?

In this function:

static void parse_and_run_stream(struct in_str *inp, int end_trigger)
{
        /* Why we need empty flag?
         * An obscure corner case "false; ``; echo $?":
         * empty command in `` should still set $? to 0.
         * But we can't just set $? to 0 at the start,
         * this breaks "false; echo `echo $?`" case.
         */
        bool empty = 1;
        while (1) {
                struct pipe *pipe_list;

                pipe_list = parse_stream(NULL, inp, end_trigger);
                if (!pipe_list) { /* EOF */
                        if (empty)
                                G.last_exitcode = 0;
                        break;
                }
                debug_print_tree(pipe_list, 0);
                debug_printf_exec("parse_and_run_stream: run_and_free_list\n");
                run_and_free_list(pipe_list);
HERE ->
                empty = 0;
        }
}

maybe it makes sense to add code (at indicated place) which reallocales
variables and functions to lower addresses, and trims malloc area.

Another thing worth trying is to make command->argv more compact.
Now it is a char**, making it occupy one contiguous block of memory
would cut down on allocations.

> >> Why would the -free- command show memory going down if the only problem
> >> is memory fragmentation?
> >
> > Because Linux caches filesystem data:
> >
> > # free
> >              total         used         free       shared      buffers
> >  Mem:      2054808      2030316        24492            0       114280
> > Swap:       131064            0       131064
> > Total:      2185872      2030316       155556
> >
> > This is on my 2 GB machine. I assure you I do *not* have nearly 2 GB
> > of programs loaded at this moment. Most of it is filesystem cache:
> >
> > shadow:~# echo 1 >/proc/sys/vm/drop_caches
> > shadow:~# free
> >              total         used         free       shared      buffers
> >  Mem:      2054808       353896      1700912            0         7564
> > Swap:       131064            0       131064
> > Total:      2185872       353896      1831976
> >
> > --
> > vda
> 
> In my case, the buffers number is always 8. Even with free memory going 
> down.

Look closer. I had only 114 MB of buffers, but the total filesystem cache
was about 1.6 GB, judging by "used" values before and after cache flush.
Buffers are only a small part of that.

--
vda