=== Kernel Sanitizers

Contact: Mark Johnston <markj@FreeBSD.org>

Sanitizers are bug detection facilities which use a combination of instrumentation inserted by the compiler (LLVM in this case) and runtime state tracking to detect bugs in C code.
They can automatically detect many types of C programming bugs, such as use-after-frees and uses of uninitialized variables, which may otherwise require substantial effort to identify.
They are particularly effective in combination with regression testing suites or fuzzing tools such as link:https://github.com/google/syzkaller[syzkaller].
Unlike tools such as Valgrind, software must be recompiled to enable a given sanitizer, but sanitizers can be used in the kernel.
Kernels with sanitizers enabled incur a significant performance overhead from the runtime, in both CPU utilization and memory usage.

Work has been ongoing to port a pair of kernel sanitizers from NetBSD to FreeBSD.
These are the Kernel Address SANitizer (KASAN) and Kernel Memory SANitizer (KMSAN).
The KASAN port is complete and has been committed to FreeBSD's development branch, and a small LLVM patch to enable KASAN and KMSAN on FreeBSD has also been committed.
KMSAN intercepts all memory accesses and verifies that they are valid using some hidden state saved for each memory allocation in the kernel; see man:kasan[9] for further details.
It can be enabled in amd64 kernels by compiling the GENERIC-KASAN kernel configuration.
The FreeBSD regression test suite currently passes with KASAN enabled.

The KMSAN port is currently in progress and is nearing completion.
KMSAN detects uses of uninitialized memory using the algorithms described in link:https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43308.pdf[the original MemorySanitizer paper].
In particular, it can detect instances of uninitialized kernel memory being copied out to userspace.
Kernel subsystems may additionally call into the KMSAN runtime to verify the state of a given buffer.
This can be used, for example, to add checks in the network stack for uninitialized bytes in egress packets.
A number of bugs have been found using KMASN and the FreeBSD regression test suite; many have already been fixed (search for "KMSAN" in src commit logs for examples), and patches have been written for all others found so far.

Future work includes:

* Finishing the KMSAN port and committing it to the FreeBSD main branch.
* Enabling CI jobs to run the test suite with KASAN and KMSAN enabled.
* Adding link:https://syzkaller.appspot.com/freebsd[syzbot] configurations with KASAN and KMSAN enabled, and fixing bugs found it.
* Reducing the scope of memory accesses which cannot be validated using KASAN or KMSAN today; this consists primarily of making use of the amd64 direct map optional in various parts of the kernel, and eliminating uses of UMA_ZONE_NOFREE.

Sponsor: The FreeBSD Foundation