Skip to content

A Wi-Fi firmware crash that took the FreeBSD kernel with it

· 6 min read

Bottom line up front

A FreeBSD 15.1 laptop rebooted on its own in the middle of the afternoon. The crash dump says page fault at address 0x10 in in6_selecthlim(), called from a TCP retransmit timer. One second earlier, the firmware on the Intel AX210 Wi-Fi card had crashed and iwlwifi(4) had detached the interface.

The firmware crash is a separate, familiar problem. The panic is a kernel race: a TCP timer looked up a route to an interface that was halfway through being removed, and read its IPv6 settings after they had already been set to NULL.

It is bug 279653, reported in June 2024 and still open. The fix proposed for main is a larger rework that does not apply to stable/15. For 15.1 there is a seven-line guard: read the pointer once and fall back to the system default hop limit if it is gone. The patch, a build script and notes are in spagu/freebsd-in6-hlim.

What the laptop logged

The laptop is a Dell XPS 13 9343 (Broadwell-U) running FreeBSD 15.1-RELEASE-p5, the same one behind the ig4 touchpad and Linuxulator posts. Its Wi-Fi card, an AX210, has a history: after suspend it sometimes stops answering with Timeout waiting for hardware access, and only a reboot brings it back.

This time the firmware went first, at 13:56:17 UTC:

iwlwifi0: Device error - SW reset
iwlwifi0: PHY ctxt cmd error. ret=-5
iwlwifi0: Failed to send LINK_CONFIG_CMD (action:3): -5
iwlwifi0: Failed to send MAC_CONFIG_CMD (action:3): -5
iwlwifi0: detached

and in the same second:

Fatal trap 12: page fault while in kernel mode
fault virtual address   = 0x10
current process         = 2 (clock (0))
panic: page fault

#5 tcp_default_output+0x1c8e
#6 tcp_timer_rexmt+0x595
#7 tcp_timer_enter+0x10c
#8 softclock_call_cc+0x16d

savecore wrote a 1.7 GB dump to /var/crash. There is no gdb on the laptop, but the message buffer inside the dump and the kernel's own disassembly are enough.

Where it faults

The instruction pointer, 0xffffffff80da9255, is inside in6_selecthlim():

callq  if_getafdata
movq   0x10(%rax), %rax     <- fault: rax is NULL
movzbl 0x1c(%rax), %eax

That is ND_IFINFO(ifp)->chlim, the hop limit an interface learned from router advertisements. ND_IFINFO() takes the interface's IPv6 data, if_afdata[AF_INET6], and reads nd_ifinfo from offset 0x10. The data pointer came back NULL, so the read went to address 0x10.

TCP passes ifp = NULL, so the function takes its last branch. It looks up the route to the peer and asks the route's interface for its hop limit:

	nh = fib6_lookup(fibnum, &dst, scopeid, 0, 0);
	if (nh != NULL) {
		hlim = ND_IFINFO(nh->nh_ifp)->chlim;
		return (hlim);
	}

The route still pointed at wlan0. wlan0 was being detached.

Why the pointer was NULL

if_detach_internal() takes an interface apart in a set order. It copies the per-protocol data aside, zeroes if_afdata[], then waits for the network epoch to drain before freeing anything. The source says what that is for:

	/*
	 * XXXZL: This net epoch wait is not necessary if we have done right.
	 * But if we do not, at least we can make a guarantee that threads those
	 * enter net epoch will see NULL address family dependent data,
	 * e.g. if_afdata[AF_INET6]. A clear NULL pointer derefence is much
	 * better than writing to freed memory.
	 */

The design accepts that a thread inside the epoch can still reach the interface and expects it to see NULL. The routing table has not dropped the interface's routes yet, so a lookup still returns a nexthop that points at it. in6_selecthlim() is one of the readers that never checks. The TCP retransmit timer was armed for a connection over IPv6 and fired in exactly that window.

The bug report has two other ways to hit the same line. One is a VNET jail whose interface is moved out while a ping is running, with a DTrace script that widens the window until it panics every time. The other is creating a bridge on 16-CURRENT. Here, the Wi-Fi firmware did the detaching.

The patch

Since the data cannot be freed while the caller is in the epoch, checking for NULL is enough. A small helper reads the pointer once and falls back to V_ip6_defhlim, the value the function already returns when there is no route at all:

+static int
+in6_ifhlim(struct ifnet *ifp)
+{
+	struct in6_ifextra *ext;
+
+	ext = if_getafdata(ifp, AF_INET6);
+	return (ext != NULL ? ext->nd_ifinfo->chlim : V_ip6_defhlim);
+}

Both branches of in6_selecthlim() use it. The ifp != NULL branch has the same exposure when a caller passes an interface that is going away. The packet built in that window goes out with the default hop limit of 64, or gets dropped further down because the interface is gone. Either outcome is better than a reboot.

Upstream, the real fix is moving towards a model where IPv6 data cannot disappear under a reader. That is D54728 and the revisions it depends on. It reworks how address families attach to interfaces and applies to main only. The guard above is a stopgap for 15.1 and does not replace that work.

Building it

This change is in the kernel itself, not in a module, so it needs a full buildkernel. Rebuilding the kernel from /usr/src also rebuilds every module, so build-kernel.sh in the repository folds in the other local fixes on this machine:

  • linux.ko and linux64.ko from the patched linux_event.c (bug 298878);
  • ig4.ko with the LPSS fixes (bug 298558).

The ig4 change has since been committed to main and merged to stable/15 and stable/14. It will ship in 15.2. It is not in any 15.1 patch level.

The script creates a boot environment first, and installkernel keeps the old kernel as /boot/kernel.old. That gives two ways back if the new kernel does not boot.

sudo sh build-kernel.sh   # patch, BE, buildkernel, installkernel
sudo shutdown -r now

It also removes an experimental atkbdc(4) resume hook left in /usr/src from the touchpad investigation. PS/2 is disabled on this laptop now, and the hook should not end up in a kernel by accident.

What is proven and what is not

The patch applies cleanly to the 15.1-p6 source tree. in6_src.c compiles with the kernel's own flags, -Werror included. That is all so far. This panic needs the firmware to crash while a TCP retransmit is pending over IPv6, and nobody can make that happen on demand. Until the patched kernel has survived a few firmware crashes, "fixed" is a claim, not a result.

The same trap as before applies too. freebsd-update will put a stock kernel back on the next base update, and the script has to run again. And none of this touches the cause: the AX210 firmware should not be crashing in the first place. That one is still under investigation.