Kernel debugging over JTAG¶

Table of contents
Kernel debugging over JTAG

As shown in the Kernel debugging with_qemu's GDB server documentation, it is easy to debug the Linux kernel in an emulated system. But some problems might only be reproducible on actual hardware (connected to the emulation setup). It is therefore sometimes necessary to debug a whole system.

JTAG is a good way to get access to the hardware state independent of the software currently running on it. It is therefore a good choice when the used hardware exposes JTAG

Requirements:

target board which exposes JTAG
- the 8devices lima board is used in this document
debug adapter hardware
- Bus Blaster v3
- Raspberry Pi's GPIO pins of the expansion header
- Bus Pirate (really slow and not recommended)
- ...
On Chip Debugger for JTAG/SWD
- OpenOCD

Preparing an debug adapter hardware¶

The connection between the debug adapter and the target board must established by connecting a couple of pins with each other. The color schema used here is from the Sparkfun Bus Pirate cable

The debug and target board via at least following of the debug adapter hardware:

TDI (Test Data In)
TDO (Test Data Out)
TCK (Test Clock)
TMS (Test Mode Select)
GND (common ground)
VTG (voltage reference)

The adapter hardware must also be selected in the OpenOCD configuration.

Only one adapter hardware must be attached. But some example configurations are shown as reference.

Bus Blaster adapter hardware¶

The Bus Blaster v3 with the 20 pin header and the default buffer logic the usage of the pins already written next to header. Just make sure to leave the board in self powered (not closed 2 pin header)

The configuration of the adapter hardware and the target board is also straight forward since the release of OpenOCD 0.11

cat > jtag_debug.cfg << "EOF" 
source [find interface/ftdi/dp_busblaster.cfg]
adapter speed 2000
transport select jtag

# in case to start debugging session without "reset halt":
# "reset halt" would be necessary here because otherwise the flash chip cannot
# be detected  and the gdb attach fails with
# "Unknown flash device (ID 0x00000000)" 
gdb_memory_map disable

source [find board/8devices-lima.cfg]
EOF

Raspberry PI (4 B) adapter hardware¶

The Raspberry PI 4B can be used as a simple adapter hardware because there are plenty of GPIOs available and OpenOCD is able to control them directly. Only the correctly raw GPIO number for each Pin has to be found. A good way to connect them (while keeping the SPI pins free for reflashing SPI chips) is:

GND: Pin 9
TDO: GPIO17 (Pin 11)
TDI: GPIO27 (Pin 13)
TCK: GPIO22 (Pin 15)
TMS: GPIO23 (Pin 16)

The hardest part is to find the correct base of the GPIO controller and to set the speed settings

The configuration of the adapter hardware and the target board is also possible since the release of OpenOCD 0.11

cat > jtag_debug.cfg << "EOF" 
adapter driver bcm2835gpio

# GPIOs for: tck tms tdi tdo
bcm2835gpio_jtag_nums 22 23 27 17

# configuration for raspberry pi 4B
bcm2835gpio_peripheral_base 0xFE000000
bcm2835gpio_speed_coeffs 236181 60

adapter speed 1000

transport select jtag

# in case to start debugging session without "reset halt":
# "reset halt" would be necessary here because otherwise the flash chip cannot
# be detected  and the gdb attach fails with
# "Unknown flash device (ID 0x00000000)" 
gdb_memory_map disable

source [find board/8devices-lima.cfg]

# allow to connect via telnet/gdb to OpenOCD from actual development machine
bindto 0.0.0.0
EOF

Bus Pirate adapter hardware¶

The Bus Pirate's SPI pins can also be used for JTAG. But it is an extremely slow debug adapter compared to the previously mentioned ones. For example a flash read_bank 0 flash.img 0 65536 takes 35682s when using the 8devices lima as target board - for example, an Raspberry Pi 4B as debug adapter hardware will only need 2 minutes for the same operations.

Just connect:

TDO: MISO
TDI: MOSI
TMS: CS
TCK: SCK
GND: GND

The configuration of the adapter hardware and the target board is also straight forward since the release of OpenOCD 0.11

cat > jtag_debug.cfg << "EOF" 
source [find interface/buspirate.cfg]

buspirate_vreg 0
buspirate_mode normal
buspirate_pullup 0

buspirate_port /dev/ttyUSB2

adapter speed 1000
transport select jtag

# in case to start debugging session without "reset halt":
# "reset halt" would be necessary here because otherwise the flash chip cannot
# be detected  and the gdb attach fails with
# "Unknown flash device (ID 0x00000000)" 
gdb_memory_map disable

source [find board/8devices-lima.cfg]
EOF

Preparing the 8devices lima target board¶

The board which should run the firmware must be connected to at least following pins of the debug adapter hardware:

TDI (Test Data In)
TDO (Test Data Out)
TCK (Test Clock)
TMS (Test Mode Select)
GND (common ground)
VTG (voltage reference)

The 8devices lima reference board exposes all over its GPIO pins:

TDI: J2 11 - GPIO1
TDO: J2 12 - GPIO2
TCK: J2 10 - GPIO0
TMS: J2 13 - GPIO3
GND: J1 16 - GND
VTG: J1 15 - 3.3V

Preparing OpenWrt¶

There is nearly no requirements from OpenWrt but there are several things which can make the debugging a lot easier.

Enable debug info¶

The actual configuration has to be set in the target kernel configuration:

CONFIG_DEBUG_INFO=y
CONFIG_DEBUG_INFO_DWARF4=y
# CONFIG_DEBUG_INFO_REDUCED is not set
CONFIG_GDB_SCRIPTS=y

The kernel address space layout randomization complicates the resolving of addresses of symbols. It is highly recommended to start the kernel with the parameter "nokaslr". For example by adding it to CONFIG_CMDLINE or by adjusting the bootargs in the bootloader. It should be checked in /proc/cmdline whether it was really booted with this parameter.

For ar71xx (8devices lima in my case), it would look like:

diff --git a/target/linux/ar71xx/config-4.14 b/target/linux/ar71xx/config-4.14
index 9a524fae4316caa10431bd6b3b4dadbe8660f14c..397e15bcecd4e9c696a2321174969541b673cbd3 100644
--- a/target/linux/ar71xx/config-4.14
+++ b/target/linux/ar71xx/config-4.14
@@ -308,10 +310,14 @@ CONFIG_CPU_SUPPORTS_MSA=y
 CONFIG_CRYPTO_RNG2=y
 CONFIG_CRYPTO_WORKQUEUE=y
 CONFIG_CSRC_R4K=y
+CONFIG_DEBUG_INFO=y
+CONFIG_DEBUG_INFO_DWARF4=y
+# CONFIG_DEBUG_INFO_REDUCED is not set
 CONFIG_DMA_NONCOHERENT=y
 CONFIG_EARLY_PRINTK=y
 CONFIG_ETHERNET_PACKET_MANGLE=y
 CONFIG_FIXED_PHY=y
+CONFIG_GDB_SCRIPTS=y
 CONFIG_GENERIC_ATOMIC64=y
 CONFIG_GENERIC_CLOCKEVENTS=y
 CONFIG_GENERIC_CMOS_UPDATE=y
diff --git a/target/linux/ar71xx/image/Makefile b/target/linux/ar71xx/image/Makefile
index 804532b55cb145134acf47accd095bbb24dee059..c485389f56c34ca8216c1016d515be2836ab2349 100644
--- a/target/linux/ar71xx/image/Makefile
+++ b/target/linux/ar71xx/image/Makefile
@@ -58,7 +58,7 @@ define Device/Default
   PROFILES = Default Minimal $$(DEVICE_PROFILE)
   MTDPARTS :=
   BLOCKSIZE := 64k
-  CONSOLE := ttyS0,115200
+  CONSOLE := ttyS0,115200 nokaslr
   CMDLINE = $$(if $$(BOARDNAME),board=$$(BOARDNAME)) $$(if $$(MTDPARTS),mtdparts=$$(MTDPARTS)) $$(if $$(CONSOLE),console=$$(CONSOLE))
   KERNEL := kernel-bin | patch-cmdline | lzma | uImage lzma
   COMPILE :=

Enabling python support for gdb¶

OpenWrt will build a gdb when CONFIG_GDB=y is set in .config. But this version is missing python support. But it can be enabled with following patch:

diff --git a/toolchain/gdb/Makefile b/toolchain/gdb/Makefile
index 41ba9853fd26d5ea2ba3759946a9591c668d92e9..afe4f01201fca21adc465a3fbd3c3751ec23df25 100644
--- a/toolchain/gdb/Makefile
+++ b/toolchain/gdb/Makefile
@@ -45,7 +45,7 @@ HOST_CONFIGURE_ARGS = \
     --without-included-gettext \
     --enable-threads \
     --with-expat \
-    --without-python \
+    --with-python \
     --disable-binutils \
     --disable-ld \
     --disable-gas \
@@ -56,9 +56,11 @@ define Host/Install
     $(INSTALL_BIN) $(HOST_BUILD_DIR)/gdb/gdb $(TOOLCHAIN_DIR)/bin/$(TARGET_CROSS)gdb
     ln -fs $(TARGET_CROSS)gdb $(TOOLCHAIN_DIR)/bin/$(GNU_TARGET_NAME)-gdb
     strip $(TOOLCHAIN_DIR)/bin/$(TARGET_CROSS)gdb
+    -$(MAKE) -C $(HOST_BUILD_DIR)/gdb/data-directory install
 endef

 define Host/Clean
+    -$(MAKE) -C $(HOST_BUILD_DIR)/gdb/data-directory uninstall
     rm -rf \
         $(HOST_BUILD_DIR) \
         $(TOOLCHAIN_DIR)/bin/$(TARGET_CROSS)gdb \

It is often possible (and in case of memory access/symbol relocation problems even recommended) to just use the normal Distro's multiarch gdb. This would be in Debian "gdb-multiarch".

Start debugging session¶

Starting OpenOCD¶

The start of OpenOCD couldn't be more trivial:

openocd -f jtag_debug.cfg

It should start a telnet server (for manual intervention) on TCP port 4444, scan the JTAG chains and afterwards start the internal gdbserver on port 3333.

Connecting gdb¶

I would use following folder in my ar71xx build environment but they will be different for other architectures or OpenWrt versions:

LINUX_DIR=${OPENWRT_DIR}/build_dir/target-mips_24kc_musl/linux-ar71xx_generic/linux-4.14.236/
GDB=${OPENWRT_DIR}/staging_dir/toolchain-mips_24kc_gcc-7.5.0_musl/bin/mips-openwrt-linux-gdb
BATADV_DIR=${OPENWRT_DIR}/build_dir/target-mips_24kc_musl/linux-ar71xx_generic/batman-adv-2019.2/

When openocd was started, we can configure gdb. It has to connect via the local openocd gdbstub to the target device. We must change to the LINUX_DIR first and can then start our target specific GDB with our uncompressed kernel image.

cd "${LINUX_DIR}" 
"${GDB}" -iex "set auto-load safe-path scripts/gdb/" -ex "target extended-remote localhost:3333"  ./vmlinux

The debug information for each module must be loaded using lx-symbols or otherwise only debug information for builtin code will be accessible

lx-symbols ..

continue

You should make sure that it doesn't load any *.ko files from ipkg-* directories. These files are stripped and doesn't contain the necessary symbol information. When necessary, just delete these folders or specify the folders with the unstripped kernel modules:

lx-symbols ../batman-adv-2019.2/.pkgdir/ ../backports-4.19.66-1/.pkgdir/ ../button-hotplug/.pkgdir/

The rest of the process works similar to debugging using gdbserver. Just set some additional breakpoints and let the kernel run again.

Some other ideas are documented in GDB Linux_snippets.

The kernel hacking debian image page should also be checked to increase the chance of getting debugable modules which didn't had all information optimized away. The relevant flags could be set directly in the routing feed like this:

diff --git a/batman-adv/Makefile b/batman-adv/Makefile
index 967965e..0abd42f 100644
--- a/batman-adv/Makefile
+++ b/batman-adv/Makefile
@@ -17,6 +17,9 @@ PKG_LICENSE_FILES:=LICENSES/preferred/GPL-2.0 LICENSES/preferred/MIT

 STAMP_CONFIGURED_DEPENDS := $(STAGING_DIR)/usr/include/mac80211-backport/backport/autoconf.h

+RSTRIP:=:
+STRIP:=:
+
 include $(INCLUDE_DIR)/kernel.mk
 include $(INCLUDE_DIR)/package.mk

@@ -77,7 +80,7 @@ define Build/Compile
         $(KERNEL_MAKE_FLAGS) \
         M="$(PKG_BUILD_DIR)/net/batman-adv" \
         $(PKG_EXTRA_KCONFIG) \
-        EXTRA_CFLAGS="$(PKG_EXTRA_CFLAGS)" \
+        EXTRA_CFLAGS="$(PKG_EXTRA_CFLAGS) -fno-inline -Og -fno-optimize-sibling-calls -fno-reorder-blocks -fno-ipa-cp-clone -fno-partial-inlining" \
         NOSTDINC_FLAGS="$(NOSTDINC_FLAGS)" \
         modules
 endef

Files (3)

jtag-busblaster.svg (27 KB) jtag-busblaster.svg		Sven Eckelmann, 07/22/2021 02:41 PM
jtag-rpi4.svg (24.1 KB) jtag-rpi4.svg		Sven Eckelmann, 07/31/2021 08:37 AM
jtag-8devices-lima.svg (25.4 KB) jtag-8devices-lima.svg		Sven Eckelmann, 08/10/2021 01:02 PM

Project

General

Profile

Open-Mesh » devtools

Wikis¶

Wiki