Greetings,
What's the canonical source for patches to 2.9BSD and 2.11BSD?
I see we have 2.11BSD patch 469 dated last month in the archive. Where does
it come from? Has anybody climbed the hill to import all the patches into a
git repo? I've found some mirrors, but moe.2bsd.org has been down for me
for ages... How does Warren keep things up to date?
I also have a (maybe faulty) memory of a similar series of patches to
2.9BSD because it was the last BSD to support non-split I&D space machines.
yet a quick google search turns up nothing other than a set of patches
dated August 1985 (also in our archive) and some changes for variants of
hardware (pro, mscp). Is that it?
Warner
Good morning all, currently trying to sort out one matter that still bewilders me with this documentation I'm working on scanning.
So I've got two copies of the "Release 5.0" User's Manual and one copy of the "System V" User's Manual. I haven't identified the exact differences, lots of pages...but they certainly are not identical, there are at least a few commands in one and not the other.
Given this, and past discussion, it's obvious Release 5.0 is the internal UNIX version that became System V, but what I'm curious about is if it was ever released publicly as "Release 5.0" before being branded as System V or if the name was System V from the moment the first commercial license was issued.
The reason I wonder this is some inconsistencies in the documentation I see out there. So both of my Release 5.0 User's Manuals have the Bell logo on the front and no mention of the court order to cease using it. Likewise, all but one of the System V related documents I received recently contain a Bell logo on the cover next to Western Electric save for the Opeartor's Guide which curiously doesn't exhibit the front page divestiture message that other documents missing the Bell logo include. Furthermore, the actual cover sheet says "Operator's Guide UNIX System Release 5.0" so technically not System V. In fact, only the User's Manual, Administrator's Manual, Error Message Manual, Transition Aids, and Release Description specifically say System V, all the rest don't have a version listed but some list Release 5.0 on their title page.
Furthering that discrepancy is this which I just purchased: https://www.ebay.com/itm/314135813726?_trkparms=amclksrc%3DITM%26aid%3D1110…
Link lives as of this sending, but contains a closed auction for an Error Message Manual from the "Release 5.0" documentation line but no Bell logo. Until the Operator's Guide and this auction link, I haven't seen any "Release 5.0" branded stuff without a Bell logo, and before I bought the System V gold set, I hadn't seen System V branded stuff *with* the Bell logo.
This shatters an assumption that I had made that at the same time the documentation branding shifted to System V was the same time the removal of the Bell logo happened, given that divestiture was what allowed them to aggressively market System V, but now this presents four distinct sets of System V gold documentation:
Release 5.0 w/ Bell logo
Release 5.0 w/o Bell logo
System V w/ Bell logo
System V w/o Bell logo
I'm curious if anyone would happen to know what the significance here is. The covers are all printed, I can't see any indication that a bunch of 5.0 manuals were retroactively painted over nor that any System V manuals got stamped with a Bell post-production. What this means is "Release 5.0" documentation was being shipped post-divestiture and "System V" was being shipped pre-divestiture. If Release 5.0 was publicly sold as System V, then what explains the post-divestiture 5.0 manuals floating around in the wild, and vice versa, if USG couldn't effectively market and support UNIX until the divestiture, how is it so many "Release 5.0" documents are floating around in well produced commercial-quality binding, both pre and post-divestiture by the time the name "System V" would've been king. Were they still maintaining an internal 5.x branch past System V that warranted its own distinct documentation set even into the commercial period? This period right around '82-'83 is incredibly fascinating and I feel very under-documented.
- Matt G.
London and Reiser report about porting the shell that “it required by far the largest conversion effort of any supposedly portable program, for the simple reason that it is not portable.” By the time of SysIII this is greatly improved, but also in porting the SysIII user land it was the most complex of the set so far.
There were three aspects that I found noteworthy:
1. London/Reiser apparently felt strongly about a property of casts. The code argues that casting an l-value should not convert it into a r-value:
<quote from "mode.h">
/* the following nonsense is required
* because casts turn an Lvalue
* into an Rvalue so two cheats
* are necessary, one for each context.
*/
union { int _cheat;};
#define Lcheat(a) ((a)._cheat)
#define Rcheat(a) ((int)(a))
<endquote>
However, Lcheat is only used in two places (in service.c), to set and to clear a flag in a pointer. Interestingly, the 32V code already replaces one of these instances with a regular r-value cast. So far, I’d never thought about this aspect of casts. I stumbled across it, because the Plan 9 compiler did not accept the Lcheat expansion as valid C.
2. On the history of dup2
The shell code includes the following:
<quote from “io.c”>
rename(f1,f2)
REG INT f1, f2;
{
#ifdef RES /* research has different sys calls from TS */
IF f1!=f2
THEN dup(f1|DUPFLG, f2);
close(f1);
IF f2==0 THEN ioset|=1 FI
FI
#else
INT fs;
IF f1!=f2
THEN fs = fcntl(f2,1,0);
close(f2);
fcntl(f1,0,f2);
close(f1);
IF fs==1 THEN fcntl(f2,2,1) FI
IF f2==0 THEN ioset|=1 FI
FI
#endif
}
<endquote>
I’ve check the 8th edition source, and indeed it supports using DUPFLG to signal to dup() that it really is dup2(). I had earlier wondered why dup2() did not appear in research until 10th edition, but now that is clear. It would seem that the dup of 8th edition is a direct ancestor to dup() in Plan 9. I wonder why this way of doing things never caught on in the other Unices.
3. Halfway to demand paging
I stumbled across this one because I had a bug in my signal handling. From early days onwards, Unix supported dynamically growing the stack allocation, which arguably is a first step towards building the mechanisms for demand paging. It appears that the Bourne shell made another step, catching page faults and expanding the data/bss allocation dynamically:
<quote from “fault.c”>
VOID fault(sig)
REG INT sig;
{
signal(sig, fault);
IF sig==MEMF
THEN IF setbrk(brkincr) == -1
THEN error(nospace);
FI
ELIF ...
<endquote>
This was already present in 7th edition, so it is by no means new in 32V or SysIII -- it had just escaped my attention as a conceptual step in the development of Unix memory handling.
Here’s a stretch, but does anybody have a copy of the 1982-ish C With
Classes Reference Manual kicking around. I can take it in n/troff or a
more modern format if you have it.
As mentioned in the first post on SysIII porting, I was surprised to see how much code was needed to initialise modern hardware and to load an OS. Of course, modern devices are much more capable than the ones of 40 years ago, so maybe my surprise is misplaced. It did raise an interest in the history of Unix system configuration though.
It would seem that 5th Edition already contained a configuration program that generated a few system tables and the ‘low.s’ file with interrupt vectors and alike. Although it steadily grew in sophistication, the approach appears still the same in SysIII. I suppose this is all in line with common practice of the era, with OS’s typically having a ’system generation kit' to combine the pre-linked OS kernel with device drivers and system tables.
SysIII also introduces the "var struct" and the “v” kernel variable that summarises some of the system configuration. I’m not sure whether it has roots in earlier Unix systems, it does not seem to originate from Research. I’m not sure what the point of this ‘v’ kernel variable was. Does anybody remember?
One could argue that one of the drivers of the success of CP/M in the 1970’s was due to its clear separation between the boot rom, BIOS and BDOS components. As far as I am aware, Unix prior to 1985 did never attempt to separate the device drivers from the other kernel code. I am not very familiar with early Xenix, it could be that Microsoft had both the skill and the interest to separate Xenix in a standard binary (i.e. BDOS part) and a device driver binary (i.e. BIOS part). Maybe the differences in MMU for the machines of the early 80’s were such that a standard binary could not be done anyway and separating out the device drivers would serve no purpose. Once the PC became dominant, maybe the point became moot for MS.
It would seem that the next step for Unix in the area of boot, config and device drivers came with Sun’s OpenBoot in 1988 or so. This also appears to be the first appearance of device trees to describe the hardware to the bios and the kernel. Moreover, it would seem to me that OpenBoot is a spiritual ancestor of the modern Risc-V SBI specification. Maybe by 1988 the IO hardware had become sufficiently complex and/or diverse to warrant a break from tradition?
Was there any other notable Unix work on better organising the boot process and the device drivers prior to OpenBoot?
> "Originally the idea of adding command line editing to ksh was
> rejected in the hope that line editing would move into the terminal
> driver." [2]
> I have always wondered, what such a central terminal driver driven
> history/line-editing would have felt like.
You can get a feel for it in Rob's "sam" editor, which works that way.
Doug
at the risk of making a fool of myself - there are several people far better qualified here, however…
my memory is that the plan9 linker could be easily rebuilt to use malloc and free in the traditional style, reducing its memory footprint - though making it much slower.
-Steve
Adam Thorton wrote:
> I mean all I really want for Christmas is a 64-bit v7 with TCP/IP support, a screen editor, and SMP support.
>
> The third one is a solved problem. The second one would not be that hard to adapt, say, uIP 0.9, to v7. That first one would require some work with C type sizes, but getting larger is easier than the reverse. It's that last one.
>
> Having said that...maybe what I really want is 64-bit 4.3 BSD?
>
> I mean, just a Unix, without all the cruft of a modern Linux, but which can actually take advantage of the resources of a modern machine. I don't care about a desktop, or even a graphical environment, I don't care about all the strange syscalls that are there to support particular databases, I don't care much about being a virtualization host.
Luther Johnson wrote:
> I'm in the process of building a system like that for myself, but
> perhaps a little smaller - mine will be based on an embedded
> microprocessor I've developed (so much work still yet to do ! at least a
> year out).
Earlier this year I ported VAX System III to Risc-V, to a simple Allwinner D1 based SBC. This is RV64GC. Just ported to the console terminal.
It turned out that porting Sys III to 64 bit was surprisingly easy, most of the kernel and user land appears to be 64 bit clean. It helps that I am using a LLP64 compiler, though. Apart from networking Sys III also feels surprisingly modern (for an ancient Unix) - its should get more attention than it does. The hardest work was in porting the VAX memory code to Risc-V page tables (and to a lesser extent, updating libc for the different FP formats).
The code is currently in an ugly state (with debug stuff in commented-out blocks, a mix of ansi and K&R styles, an incoherent kludgy build system, etc.) and the shame stopped me from putting it out on gitlab until I found enough time to clean this up. As there seems to be some interest now, I’ll put it up anyway in the next week or so. There you go Adam, half your wish comes true.
The kernel is about 60KB and most binaries are quite close in size to the VAX equivalents.
My next goals for it are to re-implement the Reiser demand paging (I think I have a good enough view of how that worked, but the proof of the pudding will be in the eating), and to add TCP/IP networking, probably the BBN stack. Making it work on RV32 and exploring early SMP work is also on my interest list.
===
David Arnold wrote:
> I think xv6 does SMP? (param.h says NCPU = 8, at least).
>
> You’d need to add a network stack and a userland, but there are options for those …
For the above, making xv6 work on the D1 board was my first stepping stone, to proof the tool chain and to get the basics right (hardware init, low-level I/O, etc.).
As an educational tool, I am sure that xv6 hits all the right spots, and it certainly does SMP (the D1 is single hart, so I have not tried that myself). I like it a lot in that context. However, as a simple Unix it is useless: from a user-land view it is less capable than LSX. At minimum it needs fixes to make the file system less constrained.
In my view, for SMP Keith Kelleman’s work for Sys-V is probably a better place to start.
Having done the SysIII 64-bit port to a recent Risc-V chip, I realised that whilst it is an interesting exercise bij itself -- and maybe even useful to students and educators around the world -- it is not ideal as a research tool for analysing Unix from the early 80’s. The address size difference adds some superfluous porting, and the 100x speed difference can hide critical algorithm constraints. Also the complex IO devices are out of character.
For a Risc-V 32 bit target I’ve settled on an FPGA implementation from the University of Tokyo. I’ve somewhat modified the system to work with the open source Yosys/NextPNR tool chain. It now implements a Linux-capable SoC with a full MMU, a 4-way cache and SD card driver in less than 4,000 lines of plain Verilog (compiling to about 14K LUTs). In a way, the code has a 6th edition feel to it: it covers a real and usable system and the code can be understood in a couple of days -- or a semester for a student who is new to the concepts involved.
https://gitlab.com/r1809/rvsoc/-/tree/main/doc
So far I have Linux and XV6 (https://gitlab.com/r1809/xv6-rv32) running, but have not started on SysIII yet.
Usefully for my use case this system is not very fast, completing an instruction in on average 10 clocks. Still, when running at 40MHz it is about 2 or 3 times as fast as a VAX11/780, which is similar to the systems of the mid-80’s. Even at this speed, a single user console Linux is surprisingly usable. By the way, funny to realise that ‘Unix/Linux capable’ has been a marketing slogan for system capability for 40 years now.
There is a short video clip with a demonstration (but running at 100MHz) here: https://youtu.be/Kt_iXVAjXcQ
Due to its simple design, the main CPU only uses some 30% of the cache memory bandwidth and it should not be all that hard to add a second CPU to the system (the CPU already supports the Risc-V atomic operations), and this could be a nice target for studying the early Unix multi-processor designs (e.g. VAX/BSD & 3B2/SVR3).
I find it an intriguing thought that the chip technology of the early 80’s (let’s say the technology used for the Bellmac-32 or M68K) would probably have sufficed to build a CPU similar to the one used in this FPGA design.
As the topic of this post is on a tangent from the focus of this list, I would recommend that any follow-ups not related to the history of Unix are sent off list.
Porting the SysIII kernel to a D1 board (https://www.aliexpress.us/item/3256803408560538.html) began with a port of XV6, in order to test the tool chain and to get comfortable with this target. Michael Engel had already ported XV6 to the D1 chip a few months before (https://www.uni-bamberg.de/fileadmin/sysnap/slides/xv6-riscv.pdf), giving a ready base to work with.
The main new effort was to add code to initialise the DRAM controller and the SD Card interface, and to have a simple boot loader. Such code is available from the manufacturer board support package (BSP), although in this case the DRAM controller code was only available as assembler compiler output and had to be reverse engineered back into C. In general I was surprised to see how big and unwieldy the BSP code is; maybe the code just looks sloppy because it has to deal with all kinds of edge cases - but I can also imagine that it accumulates cruft as it is ported from SoC to SoC by the manufacturer.
The resulting XV6 source tree is here: https://gitlab.com/pnru/xv6-d1
This version automatically boots from the SD Card on the board.
With that out of the way, the ancient Unix port was relatively easy. It would seem to me that the SysIII code base has a lot of clean-up work in it that still pays off today. The code compiles to a 64-bit target with minimal updates, which I think is a compliment to the engineers that worked on it. Probably using a LLP64 compiler also helped. In order to bring something up quickly, I modified the kernel to load ELF binaries, so that I could use proven material from the XV6 port (such as a minimalistic init and shell).
Initially, I just replaced VAX memory management with page table code taken from XV6 (i.e. no VM or swapping). Working with Risc-V page tables gives much simpler code, but I have a deeper appreciation of the VAX paging design now: for the type of code that was run in 1980, the VAX design enables very small page tables with just a few dozen entries. In contrast, for the 3-level page tables of 64-bit Risc-V I end up with 7 pages of page table of 4KB each, or 28KB -- that is larger than the memory image of many SysIII programs. If I move the ‘trampoline' to just above the stack in virtual memory it could be 5 pages instead of 7, but the overall picture remains the same. The 68020 or ‘030 MMU could be configured to have various page sizes -- this looked byzantine to me when I first saw it, but it makes more sense now.
Next I replaced the VAX scatter paging / partial swapping code, keeping the same methodology. I noticed that there is still confusion over memory management in 32V and SysIII (and implicitly SVR1,R2). The original 32V as described in the London/Reiser paper used V7 style swapping. This code can be found as ‘slowsys’ in the surviving source (https://www.tuhs.org/cgi-bin/utree.pl?file=32V/usr/src/slowsys). It was quickly (Mar vs. Jan 1979) replaced by the scatter loading / partial swapping design already hinted at in the paper (source is in 32V/usr/src/sys). Unfortunately, the “32V uses V7 swapping” meme lives on.
In scatter paging, the pages are no longer allocated continuous in physical memory but new pages are taken from a free list and expansion swaps are not usually needed. Also, when a process is swapped out, it is not fully swapped out, but just enough pages to make room for the new process. When it is readied to run again, only the partial set needs to be reloaded. In the VAX context, scatter paging and partial swapping are quite effective and I think competitive with demand paging for the 25-100KB processes that were in use at the time. As I mentioned in the post on the toolchain, the Plan 9 C compiler can easily use 1MB of memory and in a 4MB of core context, this trashes the algorithm; it starts to behave much like traditional swapping. The reason for this is that the entire process must be in memory in order to be able to run and the algorithm cannot recognise that a much smaller working set is needed. The implicit assumption of small processes can also be seen in the choice to limit partial swaps to 4KB per iteration (8 VAX pages).
For handling processes with a large memory footprint but a small working set a true demand paged VM approach is needed. The simplest such approach appears to be Richard Miller’s work for SVR1 (see June 1984 Usenix conference proceedings, "A Demand Paging Virtual Memory Manager for System V"). This is a very light touch implementation of demand paging and it seems that enough bits and pieces survive to recreate it.
The journey through the memory code made it clear again that in SysIII and before, the memory code is scattered over several locations and not so easy to fathom at first glance. It would seem that in SysV/68 an attempt was made to organise the code into separate files and with a more defined API. It does not seem to have carried through. Maybe this was because the MMU’s of the 1980-1985 era were all too different to be efficiently abstracted into a single framework.
Beyond SysV/68, were there any other attempts in the early 80’s to organise and abstract the kernel memory management code (outside BSD)?
