[TUHS] Unix install & "standalone" package

[Warner Losh]

On Mon, Sep 4, 2023 at 3:58 AM Paul Ruizendaal via TUHS <tuhs at tuhs.org>
wrote:

>
> Recently, I was looking into the “Das U-Boot” boot loader package.
> Summarised with great simplification, u-boot bundles device drivers, file
> systems, commands and a Bourne-like shell into a standalone package.
> Normally it auto-runs a script that brings up a system, but when used in
> interactive mode it allows a great deal of poking around.
>
> It made me think of the “standalone” set of programs for installing early
> Unix. On 16-bit understandably each basic command has to be a separate
> standalone program, but after the shift to 32-bit bundling more
> functionality in a single binary would have become possible.
>
> How did the Unix “standalone” package evolve in the 80’s, both in the
> research and BSD lineages? Is there any retrospective paper about that? Or
> is it a case of “Use the source, Luke”?
>

The stand package continued in research and BSD to be those programs needed
to install and/or recover
badly damaged systems.  You could create a new file system, copy a file
from the tape to a partition, etc.
You couldn't do general scripting with this, by and large.

Originally, they were tape programs. This made sense because of its
original focus. In time, some systems
could load the stand alone programs instead of the kernel, but they
continued the original focus.

This is, imho, due in large part due to the miniroot. The miniroot evolved
into both a full-enough system
to do the installation scripts in shell instead of C (Venix, at least, had
their install program written in C).
You'd copy the minroot to swap and then install the system. But a number of
additional programs were
placed into the miniroot so you could do some limited filesystem repair,
file editing, etc.

In addition, many vendor's ROMs grew in complexity. Solbourne's ROMs, for
example, could do basic
repair of UFS (clri level, not fsck level), and copy files from one place
to another. I often recovered a
Solbourne system I screwed up by attaching an external SCSI drive that had
a known good kernel,
init, etc.

The 'stand' environment was a whole set of tools that could be used to
build stand-alone programs that
shared much code of their full unix brethren, despite not having a full
kernel under them. Kernel services
were provided by different libraries that did filesystem things, block
driver things, network things, etc
in a similar way to Unix, but with a much reduced footprint.

initramfs, as has been mentioned elsewhere, is pretty much a Linux
invention. It was designed to
'punt' on the choose where to load things from and have a very minimal
interface between the boot
loader and the system. In time, it grew to support more interfaces, more
ways of loading, and better
ways to mount something that you could then 'pivot' onto. Few other unix
systems went this route, though
many adopted some variation on the pivot_root functionality. Linux has
moved beyond the pivot root after
having booted the correct kernel into being able to take over the machine
early in, say, UEFI startup with
a minimal kernel and initramfs that just knows how to load the next kernel.
They skipped the complex boot
loader stage, and went straight to the 'run linux earlier' stage which is
how things like LinuxBoot, coreboot
and others have put the boot logic into bash scripts. The ability to
'kexec' a kernel and replace the current
running kernel originated in the 'non-stop' world that wanted to reduce
downtime. Now, it's used to reduce
firmware complexity by eliminating large swaths of UEFI from the boot
process, but also generalizes in
the embedded space.

FreeBSD, from around FreeBSD 2 (1995 or so), had /rescue which largely took
over form the stand alone environment
for the repair duties of things. FreeBSD also adopted a more complex boot
loader that would load the
kernel, modules, set tunables, etc prior to kicking off the kernel. Between
/rescue having all the tools needed
to repair bad updates, repair failing disks, get that one last backup
before the drive is dead while you wait
for the new drive to be delivered, etc, and /boot/loader being able to
script loading the kernel while the BIOS
was still around so the need for drivers in the loader was lessened.
However, as the BIOS evolved into UEFI
and FreeBSD pushed into the embedded space whose firmware provided a less
rich environment to the boot
loader, so it was able to load things off fewer and fewer devices, it
became clear that it would need a pivot
root feature to allow it to boot all the way into FreeBSD, load some
drivers from an included ram disk, and then
use that mount a new root and then 'reroot' to that by killing everything
and running init from that new root.
FreeBSD also moved from Forth to Lua in its scripting language for the boot
loader, giving 'pre boot'
environment support better features. I also added the ability to use
FreeBSD boot loader as a Linux binary
to load FreeBSD and its metadata from a LinuxBoot environment. Finally,
FreeBSD has 'spun out' and
generalized the /rescue feature to allow creation of any 'BeastyBox'
environment, similar to what you get
in a busy box, or clone, environment. This environment, though, is meant in
large part on both Linux and
FreeBSD to be in constrained environments where a full install is
prohibitive (even those that never pivot
to something more, like ap, routers and nas boxes).

NetBSD retains many of the old BSD stand-alone programs that started on the
vax. I've not studied things
beyond noticing this. OpenBSD is similar. Their boot chain is a bit simpler
than FreeBSD's, though there's
noises about porting FreeBSD's boot there. There's a port of /boot/loader
to illumos too, but I don't know
if it is the default, or just available. So I'll not chat about it more.

So the original 'standalone' environment where you had one program running
on a system has evolved
into either a rich boot loader environment that lets one do a lot to decide
what kernel to load, or towards
having a minimal selection of unix programs faster and using /bin/sh or
similar to do scripting. These
reduced environments are often called standalone, though all they share
just the name with the earlier
'stand' programs: they are full unix programs, but with reduced feature
sets and 'linker magic' to package
them in a way that's faster, smaller, etc (eg all in one binary). FreeBSD's
boot loader is an outgrowth
of the original standalone env, by way of a port of NetBSD's libsa.

I suspect in the future, we'll see more and more of a trend for low-level
init and then handing off to some
built-in kernel (be it Linux, BSD-based (there's now kexec), or whatever)
to reuse more of the vetted code
rather than re-inventing Unix inside the boot loader (which is a valid
criticism of FreeBSD's boot loader,
though it's rich feature set is what you get for the complexity).

Does that answer the prompt? Should I try to make this into more of a
retrospective paper and actually
do the research on the areas I was hand-wavy about?

Warner
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.tuhs.org/pipermail/tuhs/attachments/20230904/15f8569f/attachment.htm>