[BusyBox 0003474]: nmeter crashing

bugs at busybox.net bugs at busybox.net
Sat Jun 28 23:44:20 UTC 2008


A NOTE has been added to this issue. 
====================================================================== 
http://busybox.net/bugs/view.php?id=3474 
====================================================================== 
Reported By:                nuclearcat
Assigned To:                BusyBox
====================================================================== 
Project:                    BusyBox
Issue ID:                   3474
Category:                   Other
Reproducibility:            always
Severity:                   minor
Priority:                   normal
Status:                     assigned
====================================================================== 
Date Submitted:             05-28-2008 09:22 PDT
Last Modified:              06-28-2008 16:44 PDT
====================================================================== 
Summary:                    nmeter crashing
Description: 
While trying to run nmeter on dual opteron machines

Proxy-Karam115 ~ # nmeter "CPU %c MEM %[mf] IO %b"
Segmentation fault
from their dmesg
[509567.114845] nmeter[1998]: segfault at 0 ip 08064ef2 sp bfb8f030 error
4 in busybox[8048000+76000]
another machine
[672036.057858] nmeter[6093]: segfault at 0 ip 08064ef2 sp bffffca0 error
4 in busybox[8048000+76000]

On another machine
defaulthost ~ # nmeter "CPU %c MEM %[mf] IO %b"
CPU .......... MEM 801m IO    0    0
CPU .......... MEM 801m IO    0    0
CPU .......... MEM 801m IO    0    0
CPU .......... MEM 801m IO    0    0
Segmentation fault

[917917.794357] nmeter[27440]: segfault at 0 ip 08064ef2 sp bf84d950 error
4 in busybox[8048000+76000]


BusyBox v1.10.1 (2008-04-20 17:31:36 EEST) multi-call binary

====================================================================== 

---------------------------------------------------------------------- 
 vda - 05-28-08 11:13  
---------------------------------------------------------------------- 
I tried to reproduce it with 1.10.2 and it works for me with this .config.
Can you confirn that 1.10.2 crashes for you too?

Also:

nmeter[6093]: segfault at 0 ip 08064ef2 sp bffffca0 error 4

Can you look up IP addr 08064ef2 in busybox_unstripped.map? You can find
this file in the build directory.

Look for like like these:

 .text.sortcmp  0x000000000808b694       0xac coreutils/lib.a(ls.o)
 .text.my_stat  0x000000000808b740       0x80 coreutils/lib.a(ls.o) 

---------------------------------------------------------------------- 
 nuclearcat - 05-28-08 12:27  
---------------------------------------------------------------------- 
sunfire-1 busybox-1.10.1 # cat busybox_unstripped.map |grep '08064e'
                0x0000000008064e3f       0x5c procps/lib.a(nmeter.o)
 .text.put      0x0000000008064e9b       0x38 procps/lib.a(nmeter.o)
                0x0000000008064ed3       0x2c procps/lib.a(nmeter.o)
                0x0000000008064eff       0x9f procps/lib.a(nmeter.o)

sunfire-1 busybox-1.10.1 # addr2line -i -e busybox_unstripped 08064ef2
nmeter.c:0



Also few more attempts to find which string fail:
Proxy-Karam114 ~ # nmeter "CPU %c MEM %[mf] IO %b"
Segmentation fault
Proxy-Karam114 ~ # nmeter "CPU %c MEM %[mf]"
Segmentation fault
Proxy-Karam114 ~ # nmeter "CPU %c"
CPU SUUUDDDDii
CPU SSUUUUiii.
CPU SSUUUUDDii
CPU SSUUUUDii.
CPU SSUUUUii..
CPU SSUUUUDii.
CPU SSUUUUDii.
^C
Proxy-Karam114 ~ # nmeter "CPU %c IO %b"
CPU SUUUUDii.. IO 1.9m  10m
CPU SSUUUUDii. IO 5.3m 6.0m
^C
Proxy-Karam114 ~ # nmeter "MEM %[mf]"
MEM 1.9g
MEM 1.9g
MEM 1.9g
MEM 1.9g
MEM 1.9g
^C
Proxy-Karam114 ~ # nmeter "CPU %c IO %b"
CPU SSUUUUDii. IO 2.3m 9.4m
^C
Proxy-Karam114 ~ # nmeter "CPU %c IO %b MEM %[mf]"
Segmentation fault
Proxy-Karam114 ~ # nmeter "CPU %c IO %b MEM %[mf]"
Segmentation fault

Proxy-Karam114 ~ # nmeter "CPU %c IO %b MEM %[mf]"
Segmentation fault
Proxy-Karam114 ~ # nmeter "CPU %c IO %b MEM %[mf]"
Segmentation fault
Proxy-Karam114 ~ # nmeter "CPU %c IO %b MEM %[mf] "
Segmentation fault
Proxy-Karam114 ~ # nmeter "CPU %c IO %b MEM %[mf]"
Segmentation fault
Proxy-Karam114 ~ # nmeter "CPU %c IO %b %[mf]"
Segmentation fault
Proxy-Karam114 ~ # nmeter "CPU %c %b %[mf]"
Segmentation fault
Proxy-Karam114 ~ # nmeter "%c %b %[mf]"
Segmentation fault
Proxy-Karam114 ~ # nmeter "%b %[mf]"
1.4m 288k 1.9g
4.1m 7.9m 1.9g 

---------------------------------------------------------------------- 
 vda - 05-30-08 14:55  
---------------------------------------------------------------------- 
Add a test print in procps/nmeter.c:

static void put(const char *s)
{
        int sz = strlen(s);
        if (sz > outbuf + sizeof(outbuf) - cur_outbuf)
                sz = outbuf + sizeof(outbuf) - cur_outbuf;
bb_error_msg("outbuf %p cur_outbuf %p sz %d", outbuf, cur_outbuf, sz);
        memcpy(cur_outbuf, s, sz);
        cur_outbuf += sz;
}

What does it print in your case? 

---------------------------------------------------------------------- 
 nuclearcat - 06-28-08 10:11  
---------------------------------------------------------------------- 
Sorry for delay, seems last request i didn't receive because of mailserver
failure.
Issue still actual. Here is debug output:
meter: outbuf 0x80bf990 cur_outbuf 0x80bf990 sz 4
nmeter: outbuf 0x80bf990 cur_outbuf 0x80bf994 sz 10
nmeter: outbuf 0x80bf990 cur_outbuf 0x80bf99e sz 4
nmeter: outbuf 0x80bf990 cur_outbuf 0x80bf9a2 sz 4
nmeter: outbuf 0x80bf990 cur_outbuf 0x80bf9a7 sz 4
nmeter: outbuf 0x80bf990 cur_outbuf 0x80bf9ab sz 5
nmeter: outbuf 0x80bf990 cur_outbuf 0x80bf9b0 sz 4
nmeter: outbuf 0x80bf990 cur_outbuf 0x80bf990 sz 4
nmeter: outbuf 0x80bf990 cur_outbuf 0x80bf994 sz 10
nmeter: outbuf 0x80bf990 cur_outbuf 0x80bf99e sz 4
nmeter: outbuf 0x80bf990 cur_outbuf 0x80bf9a2 sz 4
nmeter: outbuf 0x80bf990 cur_outbuf 0x80bf9a7 sz 4
nmeter: outbuf 0x80bf990 cur_outbuf 0x80bf9ab sz 5
nmeter: outbuf 0x80bf990 cur_outbuf 0x80bf9b0 sz 4
nmeter: outbuf 0x80bf990 cur_outbuf 0x80bf9b4 sz 1
CPU ii........ IO    0    0 MEM 1.9g
nmeter: outbuf 0x80bf990 cur_outbuf 0x80bf990 sz 4
nmeter: outbuf 0x80bf990 cur_outbuf 0x80bf994 sz 10
nmeter: outbuf 0x80bf990 cur_outbuf 0x80bf99e sz 4
nmeter: outbuf 0x80bf990 cur_outbuf 0x80bf9a2 sz 4
nmeter: outbuf 0x80bf990 cur_outbuf 0x80bf9a7 sz 4
nmeter: outbuf 0x80bf990 cur_outbuf 0x80bf9ab sz 5
nmeter: outbuf 0x80bf990 cur_outbuf 0x80bf9b0 sz 4
nmeter: outbuf 0x80bf990 cur_outbuf 0x80bf9b4 sz 1
CPU i......... IO    0    0 MEM 1.9g
nmeter: outbuf 0x80bf990 cur_outbuf 0x80bf990 sz 4
nmeter: outbuf 0x80bf990 cur_outbuf 0x80bf994 sz 10
nmeter: outbuf 0x80bf990 cur_outbuf 0x80bf99e sz 4
nmeter: outbuf 0x80bf990 cur_outbuf 0x80bf9a2 sz 4
nmeter: outbuf 0x80bf990 cur_outbuf 0x80bf9a7 sz 4
nmeter: outbuf 0x80bf990 cur_outbuf 0x80bf9ab sz 5
nmeter: outbuf 0x80bf990 cur_outbuf 0x80bf9b0 sz 4
nmeter: outbuf 0x80bf990 cur_outbuf 0x80bf9b4 sz 1
CPU i......... IO    0    0 MEM 1.9g
nmeter: outbuf 0x80bf990 cur_outbuf 0x80bf990 sz 4
nmeter: outbuf 0x80bf990 cur_outbuf 0x80bf994 sz 10
nmeter: outbuf 0x80bf990 cur_outbuf 0x80bf99e sz 4
nmeter: outbuf 0x80bf990 cur_outbuf 0x80bf9a2 sz 4
nmeter: outbuf 0x80bf990 cur_outbuf 0x80bf9a7 sz 4
nmeter: outbuf 0x80bf990 cur_outbuf 0x80bf9ab sz 5
nmeter: outbuf 0x80bf990 cur_outbuf 0x80bf9b0 sz 4
nmeter: outbuf 0x80bf990 cur_outbuf 0x80bf9b4 sz 1
CPU i......... IO    0    0 MEM 1.9g 

---------------------------------------------------------------------- 
 vda - 06-28-08 15:00  
---------------------------------------------------------------------- 
nmeter[6093]: segfault at 0 ip 08064ef2 sp bffffca0 error 4

.text.put 0x0000000008064e9b 0x38 procps/lib.a(nmeter.o)
                0x0000000008064ed3 0x2c procps/lib.a(nmeter.o)
                0x0000000008064eff 0x9f procps/lib.a(nmeter.o)

>From the debug printout it is obvious that put() fuction is working ok.
It's failing in a function which starts at 0x0000000008064ed3 (directly
after put()). Which function is that? Do "make procps/nmeter.s" and
"objdump -dr procps/nmeter.o" and post both results please, the function
will be visible there. 

---------------------------------------------------------------------- 
 nuclearcat - 06-28-08 16:04  
---------------------------------------------------------------------- 
Since i am using 1.11.0 (it is crashing too)
[  218.330465] nmeter[1733]: segfault at 0 ip 0806488d sp bf9d8680 error 4
in busybox[8048000+76000]

0806486e <collect_info>:
0806489a <collect_time>:

Looks like here:
Disassembly of section .text.collect_info:

00000000 <collect_info>:
   0:   53                      push   %ebx
   1:   89 c3                   mov    %eax,%ebx
   3:   83 ec 08                sub    $0x8,%esp
   6:   a1 00 00 00 00          mov    0x0,%eax
                        7: R_386_32     ptr_to_globals
   b:   80 30 01                xorb   $0x1,(%eax)
   e:   eb 14                   jmp    24 <collect_info+0x24>
  10:   8b 43 08                mov    0x8(%ebx),%eax
  13:   e8 fc ff ff ff          call   14 <collect_info+0x14>
                        14: R_386_PC32  .text.put
  18:   83 ec 0c                sub    $0xc,%esp
  1b:   53                      push   %ebx
  1c:   ff 53 04                call   *0x4(%ebx)
---->  1f:   8b 1b                   mov    (%ebx),%ebx
  21:   83 c4 10                add    $0x10,%esp
  24:   85 db                   test   %ebx,%ebx
  26:   75 e8                   jne    10 <collect_info+0x10>
  28:   59                      pop    %ecx
  29:   5b                      pop    %ebx
  2a:   5b                      pop    %ebx
  2b:   c3                      ret

i manage also to run gdb there
With disabled compiler optimizations all fine, it is not crashing.

If i enable compiler optimization:

(gdb) run nmeter "CPU %c IO %b MEM %[mf]"
Starting program: /home/root/busybox_unstripped nmeter "CPU %c IO %b MEM
%[mf]"

Program received signal SIGSEGV, Segmentation fault.
collect_info (s=0x0) at procps/nmeter.c:753
753     procps/nmeter.c: No such file or directory.
        in procps/nmeter.c

        while (s) {
                put(s->label);
                s->collect(s);
                s = s->next; <<<--- here
        }
}


(gdb) up
http://busybox.net/bugs/view.php?id=1  0x0806543c in nmeter_main (argc=2,
argv=0xbfd6b368) at
procps/nmeter.c:861
861     in procps/nmeter.c

        // Generate first samples but do not print them, they're bogus
        collect_info(first); <--- here 861
        reset_outbuf(); 

---------------------------------------------------------------------- 
 vda - 06-28-08 16:13  
---------------------------------------------------------------------- 
What does it print when you add this?

        while (s) {
                put(s->label);
                s->collect(s);
bb_error_msg("s:%p s->next:%p", s, s->next);
                s = s->next;
        } 

---------------------------------------------------------------------- 
 nuclearcat - 06-28-08 16:31  
---------------------------------------------------------------------- 
Not able to trigger bug with added line

Output:
Router-Dora ~ # ./busybox_unstripped nmeter "CPU %c IO %b MEM %[mf]"
nmeter: s:0x80c2078 s->next:0x80c20d8
nmeter: s:0x80c20d8 s->next:0x80c2100
nmeter: s:0x80c2100 s->next:(nil)
nmeter: s:0x80c2078 s->next:0x80c20d8
nmeter: s:0x80c20d8 s->next:0x80c2100
nmeter: s:0x80c2100 s->next:(nil)
CPU ii........ IO    0    0 MEM 1.9g
nmeter: s:0x80c2078 s->next:0x80c20d8
nmeter: s:0x80c20d8 s->next:0x80c2100
nmeter: s:0x80c2100 s->next:(nil)
CPU ii........ IO    0    0 MEM 1.9g
nmeter: s:0x80c2078 s->next:0x80c20d8
nmeter: s:0x80c20d8 s->next:0x80c2100
nmeter: s:0x80c2100 s->next:(nil)
CPU ii........ IO    0    0 MEM 1.9g 

---------------------------------------------------------------------- 
 vda - 06-28-08 16:38  
---------------------------------------------------------------------- 
We might have uninitialized ->next. I replaced xmalloc's with xzalloc's,
please try attached 3.patch 

---------------------------------------------------------------------- 
 nuclearcat - 06-28-08 16:44  
---------------------------------------------------------------------- 
Maybe some gcc optimization causing this?

diff in assembly of nmeter with line added and default
--- VAR1        2008-06-29 02:36:05.000000000 +0300
+++ VAR2        2008-06-29 02:37:24.000000000 +0300
@@ -5,26 +5,19 @@
    6:   a1 00 00 00 00          mov    0x0,%eax
                         7: R_386_32     ptr_to_globals
    b:   80 30 01                xorb   $0x1,(%eax)
-   e:   eb 24                   jmp    34 <collect_info+0x34>
+   e:   eb 14                   jmp    24 <collect_info+0x24>
   10:   8b 43 08                mov    0x8(%ebx),%eax
   13:   e8 fc ff ff ff          call   14 <collect_info+0x14>
                         14: R_386_PC32  .text.put
   18:   83 ec 0c                sub    $0xc,%esp
   1b:   53                      push   %ebx
   1c:   ff 53 04                call   *0x4(%ebx)
-  1f:   83 c4 0c                add    $0xc,%esp
-  22:   ff 33                   pushl  (%ebx)
-  24:   53                      push   %ebx
-  25:   68 0a 00 00 00          push   $0xa
-                        26: R_386_32    .rodata.str1.1
-  2a:   e8 fc ff ff ff          call   2b <collect_info+0x2b>
-                        2b: R_386_PC32  bb_error_msg
-  2f:   8b 1b                   mov    (%ebx),%ebx
-  31:   83 c4 10                add    $0x10,%esp
-  34:   85 db                   test   %ebx,%ebx
-  36:   75 d8                   jne    10 <collect_info+0x10>
-  38:   59                      pop    %ecx
-  39:   5b                      pop    %ebx
-  3a:   5b                      pop    %ebx
-  3b:   c3                      ret
+  1f:   8b 1b                   mov    (%ebx),%ebx
+  21:   83 c4 10                add    $0x10,%esp
+  24:   85 db                   test   %ebx,%ebx
+  26:   75 e8                   jne    10 <collect_info+0x10>
+  28:   59                      pop    %ecx
+  29:   5b                      pop    %ebx
+  2a:   5b                      pop    %ebx
+  2b:   c3                      ret
 Disassembly of section .text.collect_time:

If i remote -Os and change to -O0 it works fine
in: Makefile.flags

ifneq ($(CONFIG_DEBUG),y)
CFLAGS += $(call cc-option,-Os,) <<--- here
else 

Issue History 
Date Modified   Username       Field                    Change               
====================================================================== 
05-28-08 09:22  nuclearcat     New Issue                                    
05-28-08 09:22  nuclearcat     Status                   new => assigned     
05-28-08 09:22  nuclearcat     Assigned To               => BusyBox         
05-28-08 11:13  vda            Note Added: 0007884                          
05-28-08 12:27  nuclearcat     Note Added: 0007894                          
05-30-08 14:55  vda            Note Added: 0007904                          
06-28-08 10:11  nuclearcat     Note Added: 0008654                          
06-28-08 15:00  vda            Note Added: 0008724                          
06-28-08 16:04  nuclearcat     Note Added: 0008734                          
06-28-08 16:13  vda            Note Added: 0008744                          
06-28-08 16:31  nuclearcat     Note Added: 0008754                          
06-28-08 16:37  vda            File Added: 3.patch                          
06-28-08 16:38  vda            Note Added: 0008764                          
06-28-08 16:44  nuclearcat     Note Added: 0008774                          
======================================================================




More information about the busybox-cvs mailing list