=== PkgBase-motivated improvements to pkg

Contact: Isaac Freund <ifreund@freebsdfoundation.org>

Some problems blocking progress on the link:https://wiki.freebsd.org/PkgBase[PkgBase project] are caused by shortcomings of man:pkg[8].
The primary goal of my work on pkg is to unblock PkgBase progress.
However, all users of pkg will benefit even if they do not use PkgBase.

The scheduler for pkg's install/upgrade/delete jobs has been link:https://github.com/freebsd/pkg/pull/2330[rewritten], motivated by solving link:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=259785[PR259785].
The new scheduler models the scheduling problem as a directed graph and splits upgrade jobs into delete/install halves only when necessary to break a cycle in the graph.
This formal model gives strong guarantees about ordering that the old scheduler was not able to provide and prevents unnecessary splitting of upgrade jobs.
It also fixes longstanding bugs where the old scheduler would bail out and cause the entire upgrade to fail.
The new scheduler is included in pkg version 1.21.99.3 (pkg-devel).

The rest of my work this quarter has been related to pkg's automatic tracking of shared library dependencies, which PkgBase heavily relies on.
The initial motivating problem was link:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=265061[PR265061] but it was necessary to make more fundamental changes to how pkg tracks shlibs before cleanly solving that problem became possible.

When a package is created with man:pkg-create[8], pkg scans the included files and generates shlibs_provided/shlibs_required lists based on the executables/shared libraries found.
Before my changes, pkg would use the elf hints file of the host system as an input to pkg-create in order to filter out shlibs provided by the base system from the generated shlibs_required list.
An ALLOW_BASE_SHLIBS option disabled this filtering for the purpose of building PkgBase packages.

After my changes, pkg-create no longer reads the elf hints file of the host system and base system shlibs are included in the generated shlibs_required list.
When man:pkg-install[8]/man:pkg-upgrade[8]/etc. invoke the solver on an non-PkgBase system, pkg generates a list of shlibs provided by the base system as an input to the solver by scanning /lib and /usr/lib.
On a PkgBase system, the PkgBase packages provide all base system shlibs.

This allows the ALLOW_BASE_SHLIBS option to be eliminated.
It also gives better integration between the ports packages and PkgBase packages as shlib dependencies of ports packages on PkgBase packages are now tracked rather than ignored.
Finally, this change significantly simplifies the pkg codebase and improves portability.
This change was implemented in link:https://github.com/freebsd/pkg/pull/2386[] and is not yet included in a pkg release.

With that change and other link:https://github.com/freebsd/pkg/pull/2376[internal improvements] I was able to add support for tracking lib32 and Linuxulator shlibs, which should resolve the problem that originally motivated my work on pkg's shlib handling (link:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=265061[PR265061]).
This support is implemented in link:https://github.com/freebsd/pkg/pull/2387[] and is not yet included in a pkg release.

Sponsor: The FreeBSD Foundation
