A Wi-Fi firmware crash that took the FreeBSD kernel with it
Bottom line up front
A FreeBSD 15.1 laptop rebooted on its own in the middle of the afternoon. The
crash dump says page fault at address 0x10 in in6_selecthlim(), called
from a TCP retransmit timer. One second earlier, the firmware on the Intel
AX210 Wi-Fi card had crashed and iwlwifi(4) had detached the interface.
The firmware crash is a separate, familiar problem. The panic is a kernel race: a TCP timer looked up a route to an interface that was halfway through being removed, and read its IPv6 settings after they had already been set to NULL.
It is bug
279653, reported in
June 2024 and still open. The fix proposed for main is a larger rework that
does not apply to stable/15. For 15.1 there is a seven-line guard: read the
pointer once and fall back to the system default hop limit if it is gone. The
patch, a build script and notes are in
spagu/freebsd-in6-hlim.
What the laptop logged
The laptop is a Dell XPS 13 9343 (Broadwell-U) running FreeBSD
15.1-RELEASE-p5, the same one behind the
ig4 touchpad and
Linuxulator posts. Its Wi-Fi card,
an AX210, has a history: after suspend it sometimes stops answering with
Timeout waiting for hardware access, and only a reboot brings it back.
This time the firmware went first, at 13:56:17 UTC:
iwlwifi0: Device error - SW reset
iwlwifi0: PHY ctxt cmd error. ret=-5
iwlwifi0: Failed to send LINK_CONFIG_CMD (action:3): -5
iwlwifi0: Failed to send MAC_CONFIG_CMD (action:3): -5
iwlwifi0: detached
and in the same second:
Fatal trap 12: page fault while in kernel mode
fault virtual address = 0x10
current process = 2 (clock (0))
panic: page fault
#5 tcp_default_output+0x1c8e
#6 tcp_timer_rexmt+0x595
#7 tcp_timer_enter+0x10c
#8 softclock_call_cc+0x16d
savecore wrote a 1.7 GB dump to /var/crash. There is no gdb on the laptop,
but the message buffer inside the dump and the kernel's own disassembly are
enough.
Where it faults
The instruction pointer, 0xffffffff80da9255, is inside in6_selecthlim():
callq if_getafdata
movq 0x10(%rax), %rax <- fault: rax is NULL
movzbl 0x1c(%rax), %eax
That is ND_IFINFO(ifp)->chlim, the hop limit an interface learned from router
advertisements. ND_IFINFO() takes the interface's IPv6 data,
if_afdata[AF_INET6], and reads nd_ifinfo from offset 0x10. The data
pointer came back NULL, so the read went to address 0x10.
TCP passes ifp = NULL, so the function takes its last branch. It looks up the
route to the peer and asks the route's interface for its hop limit:
nh = fib6_lookup(fibnum, &dst, scopeid, 0, 0);
if (nh != NULL) {
hlim = ND_IFINFO(nh->nh_ifp)->chlim;
return (hlim);
}
The route still pointed at wlan0. wlan0 was being detached.
Why the pointer was NULL
if_detach_internal() takes an interface apart in a set order. It copies the
per-protocol data aside, zeroes if_afdata[], then waits for the network epoch
to drain before freeing anything. The source says what that is for:
/*
* XXXZL: This net epoch wait is not necessary if we have done right.
* But if we do not, at least we can make a guarantee that threads those
* enter net epoch will see NULL address family dependent data,
* e.g. if_afdata[AF_INET6]. A clear NULL pointer derefence is much
* better than writing to freed memory.
*/
The design accepts that a thread inside the epoch can still reach the interface
and expects it to see NULL. The routing table has not dropped the interface's
routes yet, so a lookup still returns a nexthop that points at it.
in6_selecthlim() is one of the readers that never checks. The TCP retransmit
timer was armed for a connection over IPv6 and fired in exactly that window.
The bug report has two other ways to hit the same line. One is a VNET jail
whose interface is moved out while a ping is running, with a DTrace script
that widens the window until it panics every time. The other is creating a
bridge on 16-CURRENT. Here, the Wi-Fi firmware did the detaching.
The patch
Since the data cannot be freed while the caller is in the epoch, checking for
NULL is enough. A small helper reads the pointer once and falls back to
V_ip6_defhlim, the value the function already returns when there is no route
at all:
+static int
+in6_ifhlim(struct ifnet *ifp)
+{
+ struct in6_ifextra *ext;
+
+ ext = if_getafdata(ifp, AF_INET6);
+ return (ext != NULL ? ext->nd_ifinfo->chlim : V_ip6_defhlim);
+}
Both branches of in6_selecthlim() use it. The ifp != NULL branch has the
same exposure when a caller passes an interface that is going away. The packet
built in that window goes out with the default hop limit of 64, or gets dropped
further down because the interface is gone. Either outcome is better than a
reboot.
Upstream, the real fix is moving towards a model where IPv6 data cannot
disappear under a reader. That is
D54728 and the revisions it depends on.
It reworks how address families attach to interfaces and applies to main
only. The guard above is a stopgap for 15.1 and does not replace that work.
Building it
This change is in the kernel itself, not in a module, so it needs a full
buildkernel. Rebuilding the kernel from /usr/src also rebuilds every module,
so build-kernel.sh in the repository folds in the other local fixes on this
machine:
linux.koandlinux64.kofrom the patchedlinux_event.c(bug 298878);ig4.kowith the LPSS fixes (bug 298558).
The ig4 change has since been
committed to main
and merged to stable/15 and stable/14. It will ship in 15.2. It is not in
any 15.1 patch level.
The script creates a boot environment first, and installkernel keeps the old
kernel as /boot/kernel.old. That gives two ways back if the new kernel does
not boot.
sudo sh build-kernel.sh # patch, BE, buildkernel, installkernel
sudo shutdown -r now
It also removes an experimental atkbdc(4) resume hook left in /usr/src from
the touchpad investigation. PS/2 is disabled on this laptop now, and the hook
should not end up in a kernel by accident.
What is proven and what is not
The patch applies cleanly to the 15.1-p6 source tree. in6_src.c compiles with
the kernel's own flags, -Werror included. That is all so far. This panic needs
the firmware to crash while a TCP retransmit is pending over IPv6, and nobody
can make that happen on demand. Until the patched kernel has survived a few
firmware crashes, "fixed" is a claim, not a result.
The same trap as before applies too. freebsd-update will put a stock kernel
back on the next base update, and the script has to run again. And none of this
touches the cause: the AX210 firmware should not be crashing in the first place.
That one is still under investigation.