world leader in high performance signal processing
Trace: » kgdb

KGDB: Kernel GDB

This document is for 2009R1 and newer releases.

KGDB is an in-kernel GDB stub that allows you to do source-level debugging of the kernel.

Debugging via UART

By default, KGDB communicates through a UART with a GDB client running on your host.

Since people often times only have 1 UART available on their system, KGDB will not take over the UART by default. This allows people to switch between KGDB and using the UART like normal.

To signal KGDB that you wish to start debugging with it, you send the SYSRQ command “g”. This causes the kernel to trap into debug mode and wait for a GDB connection. You can then hook up a GDB client.

Configuring

If you want to do source level debugging over UART, configure the kernel.

make linux_menuconfig

The KGDB options will be found under the Kernel Hacking section. Select pointer address check as well.

  Kernel hacking --->
    [*] Kernel debugging
    [*] KGDB: kernel debugging with remote gdb  --->
      --- KGDB: kernel debugging with remote gdb
      <*>   KGDB: use kgdb over the serial console

Don't forget to change the mode of Blackfin serial driver to PIO (not DMA). You have to enable the UART port, which you plan to connect the GDB client to. For example, if you want to run console on UART0 and connect GDB to UART1, enable both UART0 and UART1.

  Device Drivers  --->
    Character devices  --->
      Serial drivers  --->
        <*> Blackfin serial port support
        [*]   Console on Blackfin serial port
              UART Mode (PIO mode)  --->
        [*]   Enable UART0
        [*]   Enable UART1

Don't forget to unselect watchdog timer. Otherwise, kernel may be reset in debug mode.

  Device Drivers  --->
    Character devices  --->
      [*] Watchdog Timer Support  --->
        < >   Blackfin On-Chip Watchdog Timer

Modify the kernel command line

Add “kgdboc=ttyBF0,57600n8” or “kgdboc=ttyBF1,57600n8” to kernel command line to tell kernel which UART device should be used for KGDB.

if you want connect to kgdb when kernel booting. add “kgdbwait” to kernel command line.

Boot the KGDB Kernel

Boot the kernel Image.

Finally, wait for gdb connection by sending SYSRQ command “g”. (In minicom, you should enter “Ctrl+a f g” in sequence.)

Connect the Debug Host to the Target

Start GDB as for the KGDBoe session Issue the following commands at the gdb prompt

# Set the baud rate in GDB 
(gdb) set remotebaud 57600

# connect to the target
(gdb) target remote /dev/ttyS0

Debugging Kernel over Ethernet

KGDB also support debugging over ethernet on blackfin.

This allows the “normal” operation of the ethernet interface to be taken over by gdb. Selected UDP ports are then used to communicate between gdb operating on a host system and a gdb stub or server working on the target system

To set the system up you need to do the following

  • Patch the Kernel with kgdboe patch
  • Configure the kernel to build in KGDB
  • Modify the kernel command line
  • Start the Kernel
  • Start GDB on a development host
  • Connect to the target
  • Debug away

There are some advanced time saving gdb scripts, which can be used with kgdb. Check out the gdb snippets section.

Patch the kernel

A patch is available to add KGDB to the kernel.

Follow these instructions to apply the patch:

$ cd uClinux-dist/linux-2.6.x
$ patch -p0 < ../bfin_patch/kgdb_patch/kgdboe_bfin_linux-2.6.x.patch 

Configure The Kernel for KGDB

Once this patch has been applied, configure the kernel.

make linux_menuconfig

The KGDB options will be found under the Kernel Hacking section.

  Kernel hacking --->
    [*] Kernel debugging
    [*] KGBD: kernel debugging with remote gdb
      [*] KGDB: Over Ethernet

Then exit the Kernel Configuration system and save the selected options.

Modify the kernel command line

When the kernel boot the command line is passed into the kernel from u-boot in the “bootargs” environment variable or the Kernel command line can be compiled in to the kernel if required.

To trigger KGDBoe the command line is.

kgdboe=@target-IP/,@host-IP/

For example

kgdboe=@192.168.1.200/,@192.168.1.1/

This will connect the kgdb running on the target (the Blackfin board) at 192.168.1.200 to the host development system (where you compile things) at 192.168.1.1

  • The default host mac address will be used
  • The target will receive GDB messages on UDP port 6443
  • The target will send GDB messages on UDP port 6442

To compile this command line into kernel, select:

Blackfin processor options --->
      Board customizations --->
          [*] Default bootloader kernel arguments
          (console=ttyBF0,57600 kgdboe=@192.168.1.200/,@192.168.1.1/) Initial kernel command string

Boot the KGDB Kernel

Connect your terminal emulator to the serial port and boot the kernel image. The target system's IP address will need to be configured using ifconfig.

/> ifconfig eth0 192.168.1.200

Start GDB on the Host

A debug session should be started on the host system using the bfin-elf-gdb tool. Remember this runs on x86 systems but debugs Blackfin code.

cd uClinux-dist/linux-2.6.x
bfin-elf-gdb vmlinux

Or you can run ddd --debugger bfin-elf-gdb vmlinux if you want to use the ddd debugger.

Connect the Debug Host to the Target

Once the Debugger has started you can connect to the target system using the UDP protocol and the special Debug port

(gdb) target remote udp:192.168.1.200:6443

Some Sample Debug Commands

  1. These commands can be used as an example of a debug session.
    # Set a break point at a kernel function (sys_open)
    (gdb) break sys_open
    # Use the GDB continue command
    (gdb) c 
  2. On the target system use the “ls” command to trigger a breakpoint at sys_open
    /> ls
  3. The Breakpoint hits and GDB displays a message
    "Breakpoint 1: sys_open(..."
    # Use the gdb single stepping command 
    (gdb) si
    # Use the GDB  remove breakpoint command 
    (gdb) del 1
    # Set hardware breakpoint 
    (gdb) hbreak sys_open
    # Continue 
    (gdb) c<code>
      - Run "ls" in the target console <code>
    /> ls<code>
      - The GDB Session is terminated using the detach command<code>
    # Interrupt the target from the GDB host
    (gdb) Type Ctrl+C
    # Detach the GDB host from the target
    (gdb) detach
    # Exit GDB 
    (gdb) quit

Trouble shooting

"Remote communication error: Connection refused"

The ethernet connection is not established. Please check the kernel boot message (using dmesg), if network connection setting correct, you should see message like:

Kernel command line: root=/dev/mtdblock0 rw earlyprintk=seri,ua0,57600 kgdboe=@10.99.22.254/,@10.99.22.156/                
kgdboe: local port 6443                                                                                                    
kgdboe: local IP 10.99.22.254                                                                                              
kgdboe: interface eth0                                                                                                     
kgdboe: remote port 6442                                                                                                   
kgdboe: remote IP 10.99.22.156                                                                                             
kgdboe: remote ethernet address ff:ff:ff:ff:ff:ff  

"Remote failure reply: E22"

If the kernel boot message shows the network connect is OK, but gdb still cannot perform remote debug, there may be message like this:

(gdb) target remote udp:10.99.22.254:6443
warning: The remote protocol may be unreliable over UDP.
Some events may be lost, rendering further debugging impossible.
Remote debugging using udp:10.99.22.254:6443
warning: Invalid remote reply: 
Remote failure reply: E22

KGDB uses a simple network stack, if your board and host are both on a LAN, unrecognized network packet may confuse KGDB. Please connect your host with your target with cross over cable.

debugging kernel modules

Like any other ELF executable, a loadable module is divided up into several sections. The module loader looks at all of the sections and lays them out sequentially in memory; after relocating symbols it forgets about where the sections went. The kernel creates a kobject onto each loadable module and populates it with a set of attributes containing the section offsets. Those attributes will show up under /sys/module. Thus, for example, after module foo is loaded, /sys/module/foo/sections/.data will contain the beginning of the .data section. The foo developer can then fire up gdb and, after connecting to the target kernel, use the section offset information to issue a command like:

add-symbol-file /path/to/module 0xd081d000 \  # .text

-s .data 0xd08232c0 \

  1. s .bss 0xd0823e20

Thereafter, debugging the module is just like debugging the rest of the kernel. There is a script (below) which generates the add-symbol-file command, reducing the operation to a simple cut-and-paste. A little more trickery, and you could do this via rsh, as part of a gdb init script.

#!/bin/sh
#
# gdbline module image
#
# Outputs an add-symbol-file line suitable for pasting into gdb to examine
# a loaded module.
#
cd /sys/module/$1/sections
echo -n add-symbol-file $2 `/bin/cat .text`

for section in .[a-z]* *; do
    if [ $section != ".text" ]; then
        echo  " \\"
        echo -n "       -s" $section `/bin/cat $section`
    fi
done
echo

This produces output like:

root:~> ./module_helper snd_ad1836 /path/to/module
add-symbol-file /path/to/module 0x8f0000 \
        -s .bss 0x8f3d40 \
        -s .data 0x8f37e8 \
        -s .exit.text 0x8f2634 \
        -s .gnu.linkonce.this_module 0x8f3bc0 \
        -s .init.text 0x33cc780 \
        -s .rodata 0x8f2664 \
        -s .rodata.str1.4 0x8f27a0 \
        -s .strtab 0x8f32c8 \
        -s .symtab 0x8f2d98