=== DTrace: Instruction-level dynamic tracing

Links: +
link:https://wiki.freebsd.org/SummerOfCode2022Projects/InstructionLevelDynamicTracing[Wiki article] URL: link:https://wiki.freebsd.org/SummerOfCode2022Projects/InstructionLevelDynamicTracing[https://wiki.freebsd.org/SummerOfCode2022Projects/InstructionLevelDynamicTracing] +
link:https://reviews.freebsd.org/D36851[Final code review] URL: link:https://reviews.freebsd.org/D36851[https://reviews.freebsd.org/D36851] +

Contact: Christos Margiolis <christos@FreeBSD.org> +
Contact: Mark Johnston <markj@FreeBSD.org>

kinst is a new DTrace provider that allows for arbitrary kernel instruction tracing.

The provider is currently implemented only for amd64, but we plan to port it to other architectures in the future as well.

kinst probes are created on demand by libdtrace, and a probe can be created for nearly every instruction in the kernel.
Probes take the form of:

	kinst:<module>:<function>:<offset>

where "module" is the kernel module containing the named function, "function" is the kernel function to be traced, and "offset" is the offset to a specific instruction.
Omitting "offset" causes all instructions in the function to be traced.
Omitting "module" causes DTrace to search all kernel modules for the function.

For example, to trace the second instruction in amd64_syscall(), first determine the offset of the second instruction:

	# kgdb
	(kgdb) disas /r amd64_syscall
	Dump of assembler code for function amd64_syscall:
	   0xffffffff809256c0 <+0>:     55      push   %rbp
	   0xffffffff809256c1 <+1>:     48 89 e5        mov    %rsp,%rbp
	   0xffffffff809256c4 <+4>:     41 57   push   %r15

The offset is 1.
Then, to trace it:

	# dtrace -n 'kinst::amd64_syscall:1'

A new "regs" keyword was also added to the D language, providing read-only access to CPU registers at the point where the probe fired.
For example, to trace the contents of the frame pointer (register %rbp on amd64) when the kinst::amd64_syscall:1 probe fires:

	# dtrace -n 'kinst::amd64_syscall:1 {printf("0x%x", regs[R_RBP]);}'

kinst works similarly to the FBT (function boundary tracing) provider in that it places a breakpoint on the target instruction and hooks into the kernel's breakpoint handler.
It is more powerful than FBT since it can be used to create probes at arbitrary points within a function, rather than at function boundaries.
Because kinst has to be able to trace arbitrary instructions, it does not emulate most of them in software but rather causes the traced thread to execute a copy of the instruction before returning to the original code.

Planned future work includes porting kinst to additional platforms, especially arm64 and riscv, and developing tooling that can use kinst to trace calls to inline functions using the kernel's debugging symbols.

Sponsor: Google, Inc. (GSOC 2022)