world leader in high performance signal processing
Trace: » kgdb_2008r1.5

Debugging the Kernel with KGDB

This document is for 2008R1.5 release and before.

One of the features of Blackfin uClinux is the inclusion of KGDB over ethernet.

This allows the “normal” operation of the ethernet interface to be taken over by gdb. Selected UDP ports are then used to communicate between gdb operating on a host system and a gdb stub or server working on the target system

To set the system up you need to do the following

  • Patch the Kernel for use with KGDB
  • Configure the kernel to build in KGDB
  • Modify the kernel command line
  • Start the Kernel
  • Start GDB on a development host
  • Connect to the target
  • Debug away

There are some advanced time saving gdb scripts, which can be used with kgdb. Check out the gdb_snippets section.

Patch the kernel

A patch is available to add KGDB to the kernel.

Follow these instructions to apply the patch:

$ cd uClinux-dist/linux-2.6.x
$ patch -p1 < ../bfin_patch/kgdb_patch/kgdb_bfin_linux-2.6.x.patch 

Configure The Kernel for KGDB

Once this patch has been applied, configure the kernel.

make linux_menuconfig

The KGDB options will be found under the Kernel Hacking section.

  Kernel hacking --->
    [*] Kernel debugging
    [*] KGBD: kernel debugging with remote gdb
      [*] KGDB: Over Ethernet

Then exit the Kernel Configuration system and save the selected options.

Modify the kernel command line

When the kernel boot the command line is passed into the kernel from u-boot in the “bootargs” environment variable or the Kernel command line can be compiled in to the kernel if required.

To trigger KGDBoe the command line is.

kgdboe=@target-IP/,@host-IP/

For example

kgdboe=@192.168.1.200/,@192.168.1.1/

This will connect the kgdb running on the target (the Blackfin board) at 192.168.1.200 to the host development system (where you compile things) at 192.168.1.1

  • The default host mac address will be used
  • The target will receive GDB messages on UDP port 6443
  • The target will send GDB messages on UDP port 6442

To compile this command line into kernel, select:

Blackfin processor options --->
      Board customizations --->
          [*] Default bootloader kernel arguments
          (console=ttyBF0,57600 kgdboe=@192.168.1.200/,@192.168.1.1/) Initial kernel command string

Boot the KGDB Kernel

Connect your terminal emulator to the serial port and boot the kernel image. The target system's IP address will need to be configured using ifconfig.

/> ifconfig eth0 192.168.1.200

Start GDB on the Host

A debug session should be started on the host system using the bfin-elf-gdb tool. Remember this runs on x86 systems but debugs Blackfin code.

cd uClinux-dist/linux-2.6.x
bfin-elf-gdb vmlinux

Or you can run ddd --debugger bfin-elf-gdb vmlinux if you want to use the ddd debugger.

Connect the Debug Host to the Target

Once the Debugger has started you can connect to the target system using the UDP protocol and the special Debug port

(gdb) target remote udp:192.168.1.200:6443

Some Sample Debug Commands

  1. These commands can be used as an example of a debug session.
    # Set a break point at a kernel function (sys_open)
    (gdb) break sys_open
    # Use the GDB continue command
    (gdb) c 
  2. On the target system use the “ls” command to trigger a breakpoint at sys_open
    /> ls
  3. The Breakpoint hits and GDB displays a message
    "Breakpoint 1: sys_open(..."
    # Use the gdb single stepping command 
    (gdb) si
    # Use the GDB  remove breakpoint command 
    (gdb) del 1
    # Set hardware breakpoint 
    (gdb) hbreak sys_open
    # Continue 
    (gdb) c<code>
      - Run "ls" in the target console <code>
    /> ls<code>
      - The GDB Session is terminated using the detach command<code>
    # Interrupt the target from the GDB host
    (gdb) Type Ctrl+C
    # Detach the GDB host from the target
    (gdb) detach
    # Exit GDB 
    (gdb) quit

Trouble shooting

"Remote communication error: Connection refused"

The ethernet connection is not established. Please check the kernel boot message (using dmesg), if network connection setting correct, you should see message like:

Kernel command line: root=/dev/mtdblock0 rw earlyprintk=seri,ua0,57600 kgdboe=@10.99.22.254/,@10.99.22.156/                
kgdboe: local port 6443                                                                                                    
kgdboe: local IP 10.99.22.254                                                                                              
kgdboe: interface eth0                                                                                                     
kgdboe: remote port 6442                                                                                                   
kgdboe: remote IP 10.99.22.156                                                                                             
kgdboe: remote ethernet address ff:ff:ff:ff:ff:ff  

"Remote failure reply: E22"

If the kernel boot message shows the network connect is OK, but gdb still cannot perform remote debug, there may be message like this:

(gdb) target remote udp:10.99.22.254:6443
warning: The remote protocol may be unreliable over UDP.
Some events may be lost, rendering further debugging impossible.
Remote debugging using udp:10.99.22.254:6443
warning: Invalid remote reply: 
Remote failure reply: E22

KGDB uses a simple network stack, if your board and host are both on a LAN, unrecognized network packet may confuse KGDB. Please connect your host with your target with cross over cable.

Using the Serial Uart

The Serial UART can also be used to connect a debug host to the target. In this case typing a Ctrl+A over the serial line will cause the kernel to enter a debug mode. The Debug host will then send debug commands over the serial line instead of the ethernet port. This option is useful on systems that have no ethernet interfaces.

If you want to do source level debugging over UART and share this UART with console go back to the Kernel Configuration (make linux menuconfig)

Select option “KGDB: on Blackfin UART”. Set “KGDB: UART port number”. Don't forget to change the mode of Blackfin serial driver to PIO (not DMA). Or else, kgdb works incorrectly on UART

Enable “KGDB: Wait for gdb connection early” if you want connect to kgdb when kernel booting.

Exit the configuration system save the options and build the kernel.

Boot the kernel Image as above

Ask target to wait for gdb connection by entering Ctrl+A. (In minicom, you should enter Ctrl+A+A )

Start GDB as for the KGDBoe session Issue the following commands at the gdb prompt

# Set the baud rate in GDB 
(gdb) set remotebaud 57600

# connect to the target
(gdb) target remote /dev/ttyS0

Continue with debugging as before. All other operations are the same as that in KGDB over Ethernet. The only difference is that after issuing the continue command in GDB, stop the GDB connection using two “Ctrl+C”s and connect again after breakpoints are hit or Ctrl+A is entered.

debugging kernel modules

Like any other ELF executable, a loadable module is divided up into several sections. The module loader looks at all of the sections and lays them out sequentially in memory; after relocating symbols it forgets about where the sections went. The kernel creates a kobject onto each loadable module and populates it with a set of attributes containing the section offsets. Those attributes will show up under /sys/module. Thus, for example, after module foo is loaded, /sys/module/foo/sections/.data will contain the beginning of the .data section. The foo developer can then fire up gdb and, after connecting to the target kernel, use the section offset information to issue a command like:

add-symbol-file /path/to/module 0xd081d000 \  # .text

-s .data 0xd08232c0 \

  1. s .bss 0xd0823e20

Thereafter, debugging the module is just like debugging the rest of the kernel. There is a script (below) which generates the add-symbol-file command, reducing the operation to a simple cut-and-paste. A little more trickery, and you could do this via rsh, as part of a gdb init script.

#!/bin/sh
#
# gdbline module image
#
# Outputs an add-symbol-file line suitable for pasting into gdb to examine
# a loaded module.
#
cd /sys/module/$1/sections
echo -n add-symbol-file $2 `/bin/cat .text`

for section in .[a-z]* *; do
    if [ $section != ".text" ]; then
        echo  " \\"
        echo -n "       -s" $section `/bin/cat $section`
    fi
done
echo

This produces output like:

root:~> ./module_helper snd_ad1836 /path/to/module
add-symbol-file /path/to/module 0x8f0000 \
        -s .bss 0x8f3d40 \
        -s .data 0x8f37e8 \
        -s .exit.text 0x8f2634 \
        -s .gnu.linkonce.this_module 0x8f3bc0 \
        -s .init.text 0x33cc780 \
        -s .rodata 0x8f2664 \
        -s .rodata.str1.4 0x8f27a0 \
        -s .strtab 0x8f32c8 \
        -s .symtab 0x8f2d98