[TUHS] Buffer overflow found/fixed in v4 tape ;)

[Ron Natalie via TUHS]

Years ago when working at Johns Hopkins where finding ways to crash or 
otherwise compromise the UNIX system was sport, I spent a lot of time on 
the student staff there finding things and fixing them.
Buffer overruns were a frequent cause of crashes.    And it wasn’t even  
just user programs that were the problem.

There’s a quaint bug in the PDP-11 where that if you fill up your entire 
64K user address space with spl instructions, you lock up the machine.   
   Now it was a quaint exercise to the student, to write a program that 
would fill up your program (complete with overwriting the stack) to make 
that happen.   Being a lazy sort, I looked for a different solution.    
Well, our kernel had this system call called “nostack” which was used to 
get rid of UNIX’s idea of stack managment (this was done so we could 
emulate RSTS and run RSTS Basic under UNIX, the one condition that the 
department leveled on us to switch away from RSTS).

So, I turned off the UNIX stack and then tried to change the break to 
64K.   This worked … for a few minutes.
Then the machine crashed with a kernel panic with hardware errors.  This 
wasn’t the crash I was expecting (the spl gambit just locks the mahcine 
so that not even the halt key works) and I’d not even started writing 
the data.

Of course, now I had to deal with fixing the bug.    Turns out that the 
error came from the SWAP DEVICE (an RF-11 fixed head disk in our case).  
   Turns out it had found zero bytes of memory somewhere that didn’t 
exist and tried to swap 64K bytes into it.    Since the RF-11 does NPR 
(what PDP-11s call DMA), the failure is reported by the disk controller.

But more fun was had by using up other resources other than memory.     
An early UNIX security exploit was to open up all the file descriptors 
allowed and then invoke “su.”   Su figured that something must really be 
wrong if it failed to open /etc/passwd and just gave you a root shell 
anyhow.

Another fd exploit was to close certain file descriptors before invoking 
setuid programs.    I found this when fixing a user’s mounted diskpack 
which I found had the output of mount written over the superblock.    
Turns out he wanted to get rid of the message mount printed, so he just 
closed file descriptor 1.   Our mount program opened up the raw disk to 
read the pack labels (it hadn’t yet been moved to the kernel), and so 
his output got written there.   Why the disk wasn’t opened RO was just 
sloppiness.    Having found this, I launched into trying other 
user-executable set uid programs for similar bugs.    Found a efw, but 
late in the night decided to give it up and go home.

Coming in the next day I perused the (paper) logbook we kept of all 
crashes/reboot.   I found an entry that said the system was shutdown to 
restoree the accounting file that had the first 8 bytes corrupted.     
Hmm… I say.    I think I know how that happened.   The other system 
admin said, “I figured you did.   It was your user name that was the 
corrupting data (plus a colon).

-Ron