in Embedded, IOT, Linux

Embedded Linux S3C2440 – Kernel Debugging

Summary

Conduct Kernel debugging test on Embedded Linux S3C2440 by using KGDB. KGDB has been integrated into Linux kernel from version 2.6.15, it requires a serial connection to the development host. It is a cross GDB running on the development host workstation. Refer to earlier blog entry Embedded Linux S3C2440 – Kernel Module, all the tests are based on Foo.ko kernel module and test application Footest.

Startup the cross running GDB

Refer to Embedded Linux S3C2440 – Kernel Module, KGDB has been compiled into kernel and been enabled. We need to configure Embedded Linux S3C2440 to pass parameter to KGDB moudle to inform which port we use and send system request to trigger KGDB, as below steps shows,

Open Minicom and startup the Embedded Linux S3C2440 board, login as root, then do below actions,

[root@mini2440 /root]# echo ttySAC0 > /sys/module/kgdboc/parameters/kgdboc
kgdb: Registered I/O driver kgdboc.
[root@mini2440 /root]# echo g > /proc/sysrq-trigger 
SysRq : DEBUGvim
Entering KGDB

Above actions tell kernel which UART will be used for KGDB, Kgdboc means KGDB over console, so above commands shows Embedded Linux S3C2440 board kernel has entered KGDB and waiting for the connection from host GDB. KGDB is using serial link, we need to exit minicom. As below steps show,

Press Ctrl-A then Z to enter menu of minicom.
Press Q to quit without reset.
Select Yes when you have blow screen.

+----------------------+
| Leave without reset? |
|     Yes       No     |
+----------------------+

Start the GDB on host machine,

[root@localhost ~]# cd /home/iot/mini2440/linux-3.8.7/
[root@localhost linux-3.8.7]# arm-linux-gdb vmlinux
GNU gdb (GDB) 7.5.1
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http: //gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=i686-pc-linux-gnu --target=arm-buildroot-linux-uclibcgnueabi".
For bug reporting instructions, please see:
</http:><http: //www.gnu.org/software/gdb/bugs></http:>...
Reading symbols from /home/iot/mini2440/linux-3.8.7/vmlinux...done.
(gdb) 

Set the baud rate,

(gdb) set remotebaud 115200
(gdb)

Connect GDB to the S3C2440 board KGDB,

(gdb) target remote /dev/ttyUSB0
Remote debugging using /dev/ttyUSB0
kgdb_breakpoint () at kernel/debug/debug_core.c:1012
1012            arch_kgdb_breakpoint();
(gdb) step
arch_kgdb_breakpoint () at kernel/debug/debug_core.c:1011
1011            wmb(); /* Sync point before breakpoint */
(gdb) 

Above actions include, startup GDB and pass the kernel ELF file vmlinux to GDB, vmlinux is uncompressed version kernel image, connect to S3C2440 board using GDB ‘target remote’ command.

Set the break point at module.c module loading position

The driver foo.ko was compiled as a module, we need to set a break point at where the module is loaded. System call sys_init_module will copy the module information from user space to kernel space and it calls load_module(), set the break point at line 3068 and continue run kernel, as below shows.

(gdb) b kernel/module.c:3068
Breakpoint 1 at 0xc00525d4: file kernel/module.c, line 3068.
(gdb) 
(gdb) c
Continuing.

Connect Linux S3C2440 board from host through telnet.
And transfer the kernel module foo.ko and application footest from host to S3C2440 board.

[root@localhost ~]# telnet 192.168.0.11
Trying 192.168.0.11...
Connected to 192.168.0.11.
Escape character is '^]'.

mini2440 login: root
[root@mini2440 /root]#

[root@mini2440 /root]# tftp -g -r foo.ko 192.168.0.1
foo.ko               100% |********************************************************************************************************************************************| 69081   0:00:00 ETA
[root@mini2440 /root]# chmod 777 foo.ko
[root@mini2440 /root]# tftp -g -r footest 192.168.0.1
footest              100% |********************************************************************************************************************************************|  6122   0:00:00 ETA
[root@mini2440 /root]# chmod 777 footest
[root@mini2440 /root]# insmod foo.ko

After above insmod command, the foo.ko module is loaded, the GDB will stop at the break point, kernel/module.c:3068, as below gdb output shows,

[New Thread 485]
[Switching to Thread 485]

Breakpoint 1, do_init_module (mod=0xbf000344) at kernel/module.c:3068
3068                    ret = do_one_initcall(mod->init);
(gdb) step
do_one_initcall (fn=0xbf002000) at init/main.c:682
682             int count = preempt_count();
(gdb) 

ELF sections .text address from module_core and .init.text from module_init

As foo.ko is kernel module, there is no symbol information about the this module in vmlinux. We cannot set a break point at foo.ko module, and we do not know which address will the Linux kernel link to foo.ko module. Linux stores this module’s .text section address in the module information structure struct module in the module_core_element.

By using lsmod macro below, we can obtain the address of struct module allocated to foo.ko module, we will use the address retrieved to obtain the module_core address to be used as module’s .text address. And foo_init(void) function is defined with __init macro #define __init __section(.init.text) __cold notrace, this macro expands into a compiler attribute that directs the linker to place the marked portion of code into a specially named ELF section named .init.text. So the compiled foo_init() function will be placed into .init.text section of the foo.ko object module.

lsmod macro

The GDB lsmod macros in file /root/.gdbinit

#gdb List Modules Macro
define lsmod
        printf "Address\t\tModule\n"
        set $m=(struct list_head *)&modules
        set $done=0
        while ( !$done )
                # list_head is 4-bytes into struct module
                set $mp=(struct module *)((char *)$m->next - (char *)4)
                printf "0x%08X\t%s\n", $mp, $mp->name
                if ( $mp->list->next == &modules)
                        set $done=1
                end
                set $m=$m->next
        end
end

Module loading procedure debugging

When the foo.ko module is loaded, the kernel allocates a chunk of memory for the main body of the module, which is pointed to by the struct module member named module_core. It then allocates a separate chunk of memory to hold the .init.text section. After the initialization function called, the kernel frees the memory that contained the initialization function. So the object module is split, and we need to inform gdb to be able to use symbolic data for debugging the initialization function.

Below gdb commands is to add symbol file and set break points.

(gdb) lsmod
Address         Module
0xBF000344      foo
(gdb) set $m=(struct module*)0xBF000344
(gdb) p $m->module_core
$1 = (void *) 0xbf000000
(gdb) p $m->module_init
$2 = (void *) 0xbf002000
(gdb) add-symbol-file ./drivers/char/foo.ko 0xbf000000 -s .init.text 0xbf002000
add symbol table from file "./drivers/char/foo.ko" at
        .text_addr = 0xbf000000
        .init.text_addr = 0xbf002000
(y or n) y
Reading symbols from /home/iot/mini2440/linux-3.8.7/drivers/char/foo.ko...done.
(gdb) b foo_read
Breakpoint 2 at 0xbf0000a0: file drivers/char/foo.c, line 59.
(gdb) b foo_write
Breakpoint 3 at 0xbf000148: file drivers/char/foo.c, line 46.
(gdb) b do_write
Breakpoint 4 at 0xbf000100: file drivers/char/foo.c, line 28.
(gdb) 

Step into kernel module foo.ko,

(gdb) step
current_thread_info () at /home/iot/mini2440/linux-3.8.7/arch/arm/include/asm/thread_info.h:99
99              return (struct thread_info *)(sp & ~(THREAD_SIZE - 1));
(gdb) step
do_one_initcall (fn=0xbf002000 <foo_init>) at init/main.c:685
685             if (initcall_debug)
(gdb) step
682             int count = preempt_count();
(gdb) step
current_thread_info () at /home/iot/mini2440/linux-3.8.7/arch/arm/include/asm/thread_info.h:99
99              return (struct thread_info *)(sp & ~(THREAD_SIZE - 1));
(gdb) step
do_one_initcall (fn=0xbf002000 </foo_init><foo_init>) at init/main.c:685
685             if (initcall_debug)
(gdb) step
681     {
(gdb) step
685             if (initcall_debug)
(gdb) step
682             int count = preempt_count();
(gdb) step
685             if (initcall_debug)
(gdb) step
688                     ret = fn();
(gdb) step
foo_init () at drivers/char/foo.c:119
119             result = register_chrdev(FOO_MAJOR, "scull", &foo_fops);
(gdb) step
register_chrdev (fops=0x36c <foo_fops>, name=0x150 <address 0x150 out of bounds>, major=229) at drivers/char/foo.c:119
119             result = register_chrdev(FOO_MAJOR, "scull", &foo_fops);
(gdb) step
2125            return __register_chrdev(major, 0, 256, name, fops);
(gdb) step
__register_chrdev (major=major@entry=229, baseminor=baseminor@entry=0, count=count@entry=256, name=0xbf000299 "scull", name@entry=0x150 </address><address 0x150 out of bounds>, fops=0xbf0002d8, fops@entry=0x36c <foo_fops>) at fs/char_dev.c:267
267     {
(gdb) step
272             cd = __register_chrdev_region(major, baseminor, count, name);
(gdb) step
__register_chrdev_region (major=major@entry=229, baseminor=baseminor@entry=0, minorct=minorct@entry=256, name=name@entry=0xbf000299 "scull") at fs/char_dev.c:97
97      {
(gdb) step
102             cd = kzalloc(sizeof(struct char_device_struct), GFP_KERNEL);
(gdb) c
Continuing

Now step into and debug the application,

[root@mini2440 /root]# mknod /dev/foo c 229 0
[root@mini2440 /root]# ./footest
[New Thread 488]
[Switching to Thread 488]

Breakpoint 3, foo_write (filp=0xc2dc6180, buffer=0xbeaa8c70 "Hello, there", count=12, f_pos=0xc2d8ff78) at drivers/char/foo.c:46
46      {
(gdb) 
(gdb) step
50              copy_from_user(drv_buf, buffer, count);
(gdb) step
copy_from_user (n=12, from=0xbeab9c70, to=0x538 <drv_buf>) at drivers/char/foo.c:50
50              copy_from_user(drv_buf, buffer, count);

(gdb) c
Continuing.

Output of the application

[root@mini2440 /root]# mknod /dev/foo c 229 0
[root@mini2440 /root]# ./footest
The length of string buf is: 12
write Hello, there.
read ereho, tlleHsuccess!.
.
[root@mini2440 /root]#

We can see the input string was Hello, there, the expected output should be reversed string ereht ,olleH, however the output was ereho, tlleH. I think below code logic was not correct,

    for(i = 0; i<(len>>1); i++, len--)
    {
        temp = drv_buf[len-1];
        drv_buf[len-1] = drv_buf[i];
        drv_buf[i] = temp;
    }

The input len was 12. The wrong logic is as below,

i len len>>1 status
0 12 6 i = 0; i < 6
1 11 5 i = 1; i < 5
2 10 5 i = 2; i < 5
3 9 4 i = 3; i < 4

Above loop happened 4 times, it should be 6 times. half of 12, so the result was wrongly as ereho, tlleH. The correct logic should be below, and the result shall be ereht ,olleH.

i len (fixed value) len>>1 status
0 12 6 i = 0; i < 6
1 12 6 i = 1; i < 6
2 12 6 i = 2; i < 6
3 12 6 i = 3; i < 6
4 12 6 i = 4; i < 6
5 12 6 i = 5 i < 6

Following below debugging process, the loop stops when i = 4.

Breakpoint 4, do_write () at drivers/char/foo.c:28
28      {
(gdb) step
32              for(i = 0; i<(len>>1); i++, len--)
(gdb) p i
$3 = 0
(gdb) i len
Undefined info command: "len".  Try "help info".
(gdb) p $len
$4 = void
(gdb) p len
Cannot access memory at address 0x938
(gdb) step
35                      drv_buf[len-1] = drv_buf[i];
(gdb) p i
$5 = 0
(gdb) p drv_buf
Cannot access memory at address 0x538
(gdb) p drv_buf[0]
Cannot access memory at address 0x538
(gdb) p *drv_buf@10
Cannot access memory at address 0x538
(gdb) x/xw 0x538
0x538 <drv_buf>:        Cannot access memory at address 0x538
(gdb) list
30              int len = WRT_LEN;
31              char temp;
32              for(i = 0; i<(len>>1); i++, len--)
33              {
34                      temp = drv_buf[len-1];
35                      drv_buf[len-1] = drv_buf[i];
36                      drv_buf[i] = temp;
37              }
38
39      }
(gdb) p len
$6 = 12
(gdb) p *dri_buf
No symbol "dri_buf" in current context.
(gdb) p *drv_buf
Cannot access memory at address 0x538
(gdb) x/xb &drv_buf
0x538 <drv_buf>:        Cannot access memory at address 0x538
(gdb) x/xb drv_buf
0x538 <drv_buf>:        Cannot access memory at address 0x538
(gdb) x/1xb drv_buf
0x538 <drv_buf>:        Cannot access memory at address 0x538
(gdb) p drv_buf[0]
Cannot access memory at address 0x538
(gdb) p drv_buf[1]
Cannot access memory at address 0x539
(gdb) where
#0  do_write () at drivers/char/foo.c:35
#1  0xbf0001b8 in foo_write (filp=<optimized out>, buffer=<optimized out>, count=12, f_pos=<optimized out>) at drivers/char/foo.c:53
#2  0xc0096950 in vfs_write (file=file@entry=0xc2dc6180, buf=buf@entry=0xbeaa8c70 "Hello, there", count=<optimized out>, count@entry=12, pos=pos@entry=0xc2d8ff78) at fs/read_write.c:428
#3  0xc0096b24 in sys_write (fd=<optimized out>, buf=0xbeaa8c70 "Hello, there", count=12) at fs/read_write.c:475
#4  0xc000e4a0 in kuser_cmpxchg32_fixup ()
#5  0xc000e4a0 in kuser_cmpxchg32_fixup ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) whatis drv_buf
type = char [1024]
(gdb) backtrace
#0  do_write () at drivers/char/foo.c:35
#1  0xbf0001b8 in foo_write (filp=<optimized out>, buffer=<optimized out>, count=12, f_pos=<optimized out>) at drivers/char/foo.c:53
#2  0xc0096950 in vfs_write (file=file@entry=0xc2dc6180, buf=buf@entry=0xbeaa8c70 "Hello, there", count=<optimized out>, count@entry=12, pos=pos@entry=0xc2d8ff78) at fs/read_write.c:428
#3  0xc0096b24 in sys_write (fd=<optimized out>, buf=0xbeaa8c70 "Hello, there", count=12) at fs/read_write.c:475
#4  0xc000e4a0 in kuser_cmpxchg32_fixup ()
#5  0xc000e4a0 in kuser_cmpxchg32_fixup ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) p/x 0x538
$7 = 0x538
(gdb) x/i
   0x539 <drv_buf+1>:   Cannot access memory at address 0x538
(gdb) p tmp
No symbol "tmp" in current context.
(gdb) l
40
41
42
43
44
45      ssize_t foo_write(struct file *filp, const char *buffer, size_t count, loff_t *f_pos)
46      {
47              if(count > MAX_BUF_LEN)
48                      count = MAX_BUF_LEN;
49
(gdb) step
34                      temp = drv_buf[len-1];
(gdb) p temp
$8 = <optimized out>
(gdb) p len
$9 = 12
(gdb) p i
$10 = 0
(gdb) step
35                      drv_buf[len-1] = drv_buf[i];
(gdb) step
36                      drv_buf[i] = temp;
(gdb) step
32              for(i = 0; i<(len>>1); i++, len--)
(gdb) p i
$11 = 0
(gdb) step
35                      drv_buf[len-1] = drv_buf[i];
(gdb) p i
$12 = 1
(gdb) step
34                      temp = drv_buf[len-1];
(gdb) step
35                      drv_buf[len-1] = drv_buf[i];
(gdb) step
36                      drv_buf[i] = temp;
(gdb) p i
$13 = 1
(gdb) step
32              for(i = 0; i<(len>>1); i++, len--)
(gdb) step
35                      drv_buf[len-1] = drv_buf[i];
(gdb) step
34                      temp = drv_buf[len-1];
(gdb) step
35                      drv_buf[len-1] = drv_buf[i];
(gdb) step
36                      drv_buf[i] = temp;
(gdb) step
^[[A32          for(i = 0; i<(len>>1); i++, len--)
(gdb) step
35                      drv_buf[len-1] = drv_buf[i];
(gdb) p
$14 = 1
(gdb) p i
$15 = 3
(gdb) p i
$16 = 3
(gdb) p
$17 = 3
(gdb) step
34                      temp = drv_buf[len-1];
(gdb) step
35                      drv_buf[len-1] = drv_buf[i];
(gdb) step
36                      drv_buf[i] = temp;
(gdb) step
32              for(i = 0; i<(len>>1); i++, len--)
(gdb) step
39      }
(gdb) p i
$18 = 4
(gdb) step
foo_write (filp=<optimized out>, buffer=<optimized out>, count=12, f_pos=<optimized out>) at drivers/char/foo.c:55
55      }
(gdb) p len
No symbol "len" in current context.
(gdb) c
Continuing.

Breakpoint 2, foo_read (filp=0xc2dc6180, buffer=0xbeaa8c70 "Hello, there", count=100, ppos=0xc2d8ff78) at drivers/char/foo.c:59
59      {
(gdb) c
Continuing.

The serial port output was like below,

foo initialized.
device open success!.
user write data to driver.
user read data from driver.
Device release.
device open success!.
user write data to driver.
user read data from driver.
Device release.

Cannot access memory at address

Above debugging process has one Cannot access memory at address error, refer to Linux Device Drivers, Second Edition by Alessandro Rubini, Jonathan Corbet, I guess it was the reason.

Note that you can’t disassemble a module function, because the debugger is acting on vmlinux, which doesn’t know about your module. If you try to disassemble a module by address, gdb is most likely to reply “Cannot access memory at xxxx.” For the same reason, you can’t look at data items belonging to a module. They can be read from /dev/mem if you know the address of your variables, but it’s hard to make sense out of raw data extracted from system RAM.

Reference

The 101 of ELF files on Linux: Understanding and Analysis
Debugging The Linux Kernel Using Gdb
Embedded Linux S3C2440 – Kernel Module
Linux Device Drivers, Second Edition by Alessandro Rubini, Jonathan Corbet

Write a Comment

Comment