29 Mar
2020
29 Mar
'20
12:27 p.m.
Here is a question for the old hands from the Labs, I’m trying to get the timeline of some
development steps right.
The two main things are: when did the 4.1 merge take place, and when were ‘streams’
added?
Going by file dates, the surviving 8th edition source appears to be from 1985. I can see
that a lot of files in /usr/include did not change after Jan 1982 (e.g. nlist.h). This
suggests that early in 1982 the merge between 4.1 code and 32V code took place, to create
the foundation for further development (“proto 8th edition”, so to speak).
Similarly, there are a dozen or so files in the kernel that all have a file date of
November 1982. The most interesting one of these is “dtline.c”, a character mode Datakit
driver: it uses ‘streams’. This suggests that there was a further code merge late in 1982
and implies that ‘streams’ were developed prior to that date.
From the S/F-Unix papers it seems that ‘streams’ did not exist in 1981, at least they are
not mentioned in an otherwise comprehensive set of papers. On the other hand, the S/F-Unix
work was done in the Exploratory group, not the Research group: maybe it was inappropriate
to mention.
All in all, my hypotheses would be that:
- the 32V/4.1 merge took place early in 1982
- ‘streams’ were developed in 1982 on 32V (maybe also V7) systems
- a further merge took place late in 1982 that combined the new base with latest
developments
Does that sound correct, or was it all different?
Related is the question when the "file system switch" was added. It must have
been later than 1981 and before 1985, but I have not been able to pinpoint it further.
Paul
29 Mar
29 Mar
2:04 p.m.
Paul Ruizendaal <pnr(a)planet.nl> wrote:
Related is the question when the "file system
switch" was added. It must
have been later than 1981 and before 1985, but I have not been able to
pinpoint it further.
IIRC there was a "paper" (only an abstract) on the file system
switch published in a USENIX conference proceedings. That woud help
trace it down.
Peter Weinberger, who did it, is at Google; you could ask him
directly, as well.
Arnold
6:12 p.m.
On 29 Mar 2020, at 16:04, arnold(a)skeeve.com wrote:
Paul Ruizendaal <pnr(a)planet.nl> wrote:
I have that paper (“The Unix 8th Edition Network File System”), it was presented at a
March 1985 ACM conference. However, there are indications that the roots of the file
system switch existed earlier, possibly much earlier.
I think Doug McIlroy once described 1973 as a pivotal year for Unix, with many concepts
devised that would blossom in the following 3-5 years. I’m increasingly tempted to think
that Summer ’81 - Summer ’82 was a similarly pivotal year.
Related is the question when the "file
system switch" was added. It must
have been later than 1981 and before 1985, but I have not been able to
pinpoint it further.
IIRC there was a "paper" (only an abstract) on the file system
switch published in a USENIX conference proceedings. That woud help
trace it down. Peter Weinberger, who did it, is at Google; you could
ask him
directly, as well.
That is a good idea. If someone has the email address I’d appreciate an off list message.
Paul
9:48 p.m.
I have a dog in the fight, having joined in June 1980, but that is not
a coincidence. The period of 1980-1982 was a big one for 127 (soon
1127) as they were finally given the chance to grow, and I was one of
the lucky early hires in that burst. New blood brought in new ideas
and things happened fast.
It was also the time of the VAX; the center's 11/780 arrived in late
1980 I think, maybe early 1981. Our first experiments with graphical
terminals spanned 1980 to early 1981, using Greg Chesson's mux, but by
late 1981 we were using Dennis's streams (only STREAMS when they went
to USG) and the select system call, which was by then running in a
merged Berkeley/Research Unix that eventually became the Eighth
Edition.
My notebooks can probably lock down a lot of this as I was a prolific
note-taker back then, when they still made paper.
-rob
On Mon, Mar 30, 2020 at 5:14 AM Paul Ruizendaal <pnr(a)planet.nl> wrote:
On 29 Mar 2020, at 16:04, arnold(a)skeeve.com wrote:
Paul Ruizendaal <pnr(a)planet.nl> wrote:
I have that paper (“The Unix 8th Edition Network File System”), it was presented at a
March 1985 ACM conference. However, there are indications that the roots of the file
system switch existed earlier, possibly much earlier.
I think Doug McIlroy once described 1973 as a pivotal year for Unix, with many concepts
devised that would blossom in the following 3-5 years. I’m increasingly tempted to think
that Summer ’81 - Summer ’82 was a similarly pivotal year.
Related is the question when the "file
system switch" was added. It must
have been later than 1981 and before 1985, but I have not been able to
pinpoint it further.
IIRC there was a "paper" (only an abstract) on the file system
switch published in a USENIX conference proceedings. That woud help
trace it down. Peter Weinberger, who did it, is at Google; you
could ask him
directly, as well.
That is a good idea. If someone has the email address I’d appreciate an off list
message.
Paul
7:43 a.m.
On 29 Mar 2020, at 23:48, Rob Pike
<robpike(a)gmail.com> wrote:
I have a dog in the fight, having joined in June 1980, but that is not
a coincidence. The period of 1980-1982 was a big one for 127 (soon
1127) as they were finally given the chance to grow, and I was one of
the lucky early hires in that burst. New blood brought in new ideas
and things happened fast.
I had not realised that the Research group expanded in 1980, but it fits.
It was also the time of the VAX; the center's
11/780 arrived in late
1980 I think, maybe early 1981.
I did realise that bit, and it made me wonder if the ’73 burst was in part driven by the
arrival of a 11/45.
Our first experiments with graphical terminals spanned
1980 to early 1981,
Yes, as you may remember from this list I dove into that last November - schematics, tools
& firmware.
using Greg Chesson's mux,
Chesson’s MPX files remain a puzzle piece that is somewhat difficult to fit in the overall
story, having so many aspects. It sits between Rand ports and SysIII fifo’s, experiments
with non-blocking I/O, has aspects of pseudo-terminals, etc. I have not been able to
figure out what immediate need they served, unless it was used in the first generation
Datakit software (as MPX precedes the Jerq, that cannot have been the immediate need.)
but by late 1981 we were using Dennis's streams
(only STREAMS when
they went to USG) and the select system call, which was by then running
in a merged Berkeley/Research Unix that eventually became the Eighth
Edition.
To be honest, late 1981 sounds a bit too early for the merge. The 4.1 code was ready in
June 1981 and the ’select’ system call was first proposed in July 1981, so it is possible.
However, in the BSD line ’select’ was not fully implemented until March/April 1982.
It is certainly possible, even likely, that ‘streams’ date from 1981 or earlier. Networks
don’t mesh well with TTY line disciplines and clist buffering - that pain will have become
apparent already in 1979. Maybe it was among the first things to be fixed when the VAX
arrived.
My notebooks can probably lock down a lot of this as I
was a prolific
note-taker back then, when they still made paper.
If someday you have time for this, it would be much appreciated!
Paul
9:06 a.m.
I've looked through my notes and unfortunately there's very little
about this as the notes are mostly about graphics and physics.
You're right I was a little early above, but the "merge" was not an
event. It was a process that went on for months, years even.
Dennis wrote the select we used, I'm almost certain of that. We (he
and I) talked about its design at the time, after a trip he made to
Berkeley about the DARPA Stuff. He had had many discussions with the
BSD work going on and wanted to couple it to the streams work to make
my graphical multiplexer work. Mpx was killing me (and killing the
kernel), but I'm nearly certain that I was using select (with our form
of pseudo-ttys) before or by very early 1982. I'm all but certain; we
gave up on mpx(2) very early.
Streams came later; Dennis's Show and Tell about them was early in 1982.
I'm not sure, but we may have booted London & Reiser's VAX port first,
but we moved to Berkeley code pretty early. We (Dennis mostly, but
others) were talking to Berkeley often. We didn't just wait for a
release; we were a part (not sure I could say now how important, but a
part) of it from early on.
-rob
On Mon, Mar 30, 2020 at 6:44 PM Paul Ruizendaal <pnr(a)planet.nl> wrote:
On 29 Mar 2020, at 23:48, Rob Pike
<robpike(a)gmail.com> wrote:
I have a dog in the fight, having joined in June 1980, but that is not
a coincidence. The period of 1980-1982 was a big one for 127 (soon
1127) as they were finally given the chance to grow, and I was one of
the lucky early hires in that burst. New blood brought in new ideas
and things happened fast.
I had not realised that the Research group expanded in 1980, but it fits.
It was also the time of the VAX; the
center's 11/780 arrived in late
1980 I think, maybe early 1981.
I did realise that bit, and it made me wonder if the ’73 burst was in part driven by the
arrival of a 11/45.
Our first experiments with graphical terminals
spanned 1980 to early 1981,
Yes, as you may remember from this list I dove into that last November - schematics,
tools & firmware.
using Greg Chesson's mux,
Chesson’s MPX files remain a puzzle piece that is somewhat difficult to fit in the
overall story, having so many aspects. It sits between Rand ports and SysIII fifo’s,
experiments with non-blocking I/O, has aspects of pseudo-terminals, etc. I have not been
able to figure out what immediate need they served, unless it was used in the first
generation Datakit software (as MPX precedes the Jerq, that cannot have been the immediate
need.)
but by late 1981 we were using Dennis's
streams (only STREAMS when
they went to USG) and the select system call, which was by then running
in a merged Berkeley/Research Unix that eventually became the Eighth
Edition.
To be honest, late 1981 sounds a bit too early for the merge. The 4.1 code was ready in
June 1981 and the ’select’ system call was first proposed in July 1981, so it is possible.
However, in the BSD line ’select’ was not fully implemented until March/April 1982.
It is certainly possible, even likely, that ‘streams’ date from 1981 or earlier. Networks
don’t mesh well with TTY line disciplines and clist buffering - that pain will have become
apparent already in 1979. Maybe it was among the first things to be fixed when the VAX
arrived.
My notebooks can probably lock down a lot of this
as I was a prolific
note-taker back then, when they still made paper.
If someday you have time for this, it would be much appreciated!
Paul
1:26 p.m.
Paul - see below..
On Mon, Mar 30, 2020 at 3:43 AM Paul Ruizendaal <pnr(a)planet.nl> wrote:
To be honest, late 1981 sounds a bit too early for the
merge. The 4.1 code
was ready in June 1981 and the ’select’ system call was first proposed in
July 1981, so it is possible. However, in the BSD line ’select’ was not
fully implemented until March/April 1982.
Please be careful here. 4.1BSD is different from the pre-'4.2' released
and
4.2BSD itself. 4.1aBSD was the first pseudo release[1] that started to have
the major surgery to support Bill's sockets idea and splice in the UCB
rewrite of the BBN code. 4.1BSD was the first system for the Vax that
really wide distribution. 'Anyone' with an AT&T license could get it and
most people did. Remember this is the system that BBN (Gurwitz) did the
original IP/TCP support (sans sockets - i.e. /dev/ip /dev/tcp ...). Your
date of June '81 for the 4.1BSD release seems late, but I'll accept it.
3BSD was 1979, and I thought 4BSD was a year later, with 4.1BSD a few
months after 4BSD (few people actually got 4BSD)
That said, Bill and Sam did the heavy lifting on select(2) first in 4.1aBSD
and there were some issues (again I have forgotten the details -- I do
remember, I was working on my thesis and I had a do a huge rewrite of the
AP kernel support to handle select(2) properly). I remember talking to Sam
(arguing with him most probably) about it one night before it was fully
created. I want to say, he had worked on something similar at the firm he
was at (the firm name I now forget -- si-mumble -- they were in Mt. View)
before he joined CRSG. I don't remember now the issues I had, but I do
remember it was a bit of mess to support the way the AP hardware assumed it
could do DMA on the UBA[2]
[1] 4.1a/4.1b/4.1cBSD was officially internal to UCB and some ARPA-sites,
although I would have expected someone like Dennis in 1127 to have been
sent it also, as wnj was in the process of leaving for Sun and he took them
with him. For instance, I would take 4.1c to Masscomp. The key is that
these were not as widely distributed as 4.1BSD. 4.2BSD would really
accelerate BSD UNIX uptake, because of the networking support but there was
more than 2-3 years between 4.1BSD and 4.2BSD.
[2] The AP's MMU/DMA interface at the time, was causing me great hair, and
that was likely to have been part of the reason I wanted some help/changes
in the KPI interface - which is actually funny, they eventually came with
the CMU Mach MMU changes of 4.4BSD (which was much more friendly to a
multiprocessor/coprocessor architecture). FWIW: I never got them when I
was at UCB.
1:38 p.m.
On Mon, 30 Mar 2020, Clem Cole wrote:
...
kernel support to?handle?select(2) properly).? I
remember talking to Sam (arguing
with him most probably) about it one night before it was fully created.? ?Iwant
to say, he had worked on something similar at the firm he was at (the firm
name I now forget -- si-mumble -- they were in Mt. View) before he joined
CRSG.? I don't remember now the issues I had, but I do remember it was a bit
of mess to support the way the AP hardware assumed it could do DMA on the
UBA[2]
From my notes:
After graduating, Leffler got a job in the Silicon Valley
at Sytek, a small firm in Sunnyvale.
% CITE: archives/csrg-archives/disk1/mnt/2.79/usenet/netdir/sites/sytek
They used a PDP 11/45 and moved to a VAX 11/750 running VM/UNIX
with four 300 baud ACU's (Automatic Call Units aka ``modems``) ---
they marketed a broadband network to
connect thousands of stations using coaxial cable over a distance
of 40 miles.
% maybe CITE:
http://books.google.com/books?id=AyZfLaxYr9YC&pg=PA43&lpg=PA43&…
%28408%29+734-9000++sytek&source=bl&ots=SYGI9rBAOV&sig=rxBo1SgfqdTLuNotXaa59nrR6
C4&hl=en&sa=X&ei=CzbwUJTjI6HN2QX48oGIAw&ved=0CEAQ6AEwBA#v=onepage&q=%28408%29%20
734-9000%20%20sytek&f=false
Since they were using a VAX and he had kept in touch with Joy, he was
a beta-test site for 4BSD.\cite{salus1994}
% NOTE: about sytek: an ad to find employees
%http://books.google.com/books?id=ChMAmfS1nEkC&pg=PA85&lpg=PA85&dq=%28408%29+734
-9000++sytek&source=bl&ots=aS_yQznXAy&sig=6vA7NTbspu3gcf_jT2sOiUSRPZU&hl=en&sa=X
&ei=CzbwUJTjI6HN2QX48oGIAw&ved=0CC4Q6AEwAA#v=onepage&q=%28408%29%20734-9000%20%2
0sytek&f=false
% NOTE: not to take over since no Sun then
Leffler went to Berkeley to work with Joy\cite{billshannon2} in the
fall of 1981.\cite{salus1994}.
2:41 p.m.
Jeremy -- Sytek - that was it -- thank you. Sunnyvale/Mt. View -- who can
tell the difference? ;-)
Thanks again,
Clem
On Mon, Mar 30, 2020 at 10:21 AM <reed(a)reedmedia.net> wrote:
On Mon, 30 Mar 2020, Clem Cole wrote:
...
kernel support to?handle?select(2) properly).? I
remember talking to Sam
(arguing
with him most probably) about it one night before
it was fully created.?
?Iwant
to say, he had worked on something similar at the
firm he was at (the
firm
name I now forget -- si-mumble -- they were in
Mt. View) before he joined
CRSG.? I don't remember now the issues I had, but I do remember it was a
bit
of mess to support the way the AP hardware
assumed it could do DMA on the
UBA[2]
From my notes:
After graduating, Leffler got a job in the Silicon Valley
at Sytek, a small firm in Sunnyvale.
% CITE: archives/csrg-archives/disk1/mnt/2.79/usenet/netdir/sites/sytek
They used a PDP 11/45 and moved to a VAX 11/750 running VM/UNIX
with four 300 baud ACU's (Automatic Call Units aka ``modems``) ---
they marketed a broadband network to
connect thousands of stations using coaxial cable over a distance
of 40 miles.
% maybe CITE:
http://books.google.com/books?id=AyZfLaxYr9YC&pg=PA43&lpg=PA43&…
%28408%29+734-9000++sytek&source=bl&ots=SYGI9rBAOV&sig=rxBo1SgfqdTLuNotXaa59nrR6
C4&hl=en&sa=X&ei=CzbwUJTjI6HN2QX48oGIAw&ved=0CEAQ6AEwBA#v=onepage&q=%28408%29%20
734-9000%20%20sytek&f=false
<http://books.google.com/books?id=AyZfLaxYr9YC&pg=PA43&lpg=PA43&dq=%28408%29+734-9000++sytek&source=bl&ots=SYGI9rBAOV&sig=rxBo1SgfqdTLuNotXaa59nrR6C4&hl=en&sa=X&ei=CzbwUJTjI6HN2QX48oGIAw&ved=0CEAQ6AEwBA#v=onepage&q=%28408%29%20734-9000%20%20sytek&f=false>
Since they were using a VAX and he had kept in touch with Joy, he was
a beta-test site for 4BSD.\cite{salus1994}
% NOTE: about sytek: an ad to find employees
%
http://books.google.com/books?id=ChMAmfS1nEkC&pg=PA85&lpg=PA85&…
-9000++sytek&source=bl&ots=aS_yQznXAy&sig=6vA7NTbspu3gcf_jT2sOiUSRPZU&hl=en&sa=X
&ei=CzbwUJTjI6HN2QX48oGIAw&ved=0CC4Q6AEwAA#v=onepage&q=%28408%29%20734-9000%20%2
0sytek&f=false
<http://books.google.com/books?id=ChMAmfS1nEkC&pg=PA85&lpg=PA85&dq=%28408%29+734-9000++sytek&source=bl&ots=aS_yQznXAy&sig=6vA7NTbspu3gcf_jT2sOiUSRPZU&hl=en&sa=X&ei=CzbwUJTjI6HN2QX48oGIAw&ved=0CC4Q6AEwAA#v=onepage&q=%28408%29%20734-9000%20%20sytek&f=false>
% NOTE: not to take over since no Sun then
Leffler went to Berkeley to work with Joy\cite{billshannon2} in the
fall of 1981.\cite{salus1994}.
3:06 p.m.
New subject: Sytek (was Re: 8th Edition timeline)
On 3/30/2020 7:41 AM, Clem Cole wrote:
Jeremy -- Sytek - that was it -- thank you.
Sunnyvale/Mt. View -- who
can tell the difference? ;-)
Here's a story about Sytek (I hope I haven't told it before - I'm
starting to forget who I've told stories to).
In 1977/1978 I was an extremely junior engineeer at Ford Aerospace.
I was in the group that was both acting as the main Unix promoter
in Ford, and also was making a provably-secure version of Unix
called KSOS. The group had a bunch of excellent senior people (e.g.
John Nagle, Tom Berson, Ken Biba, Mike Pliner, Mike Padlipsky).
This was my first real job in the software industry so I didn't know
what to expect. After about 6 months I noticed that there were a lot
of closed-door meetings going on. I thought this was normal. One day
I was walking by an office when the door opened up and I was invited
in. I didn't know if I had done something wrong and was about to be
fired, or what.
I turned out that I had nothing to worry about. What was happening
was that some of the people in the group were going to leave Ford
to start their own company. They needed some junior people also,
and they asked me to join them. I gave it some thought but turned
them down because I didn't want to live in Silicon Valley anymore,
and wanted to return to Santa Barabara where I belonged.
The company that they started was Sytek, which got the contract
from IBM to create NetBIOS. They also did other networking work
that has been mentioned in this thread.
If I had gone to Sytex I would have been employe #8 with lots of
stock options. I often wonder how my life would have been different
if I had done this. I suspect many people on TUHS have similar stories
about the ones that got away. In fact, this might be a good topic
for a new thread.
Cordially,
Jon Forrest
4:02 p.m.
Comments in line:
On 30 Mar 2020, at 15:26, Clem Cole
<clemc(a)ccc.com> wrote:
Paul - see below..
On Mon, Mar 30, 2020 at 3:43 AM Paul Ruizendaal <pnr(a)planet.nl> wrote:
To be honest, late 1981 sounds a bit too early for the merge. The 4.1 code was ready in
June 1981 and the ’select’ system call was first proposed in July 1981, so it is possible.
However, in the BSD line ’select’ was not fully implemented until March/April 1982.
Please be careful here. 4.1BSD is different from the
pre-'4.2' released and 4.2BSD itself. 4.1aBSD was the first pseudo release[1]
that started to have the major surgery to support Bill's sockets idea and splice in
the UCB rewrite of the BBN code. 4.1BSD was the first system for the Vax that really wide
distribution. 'Anyone' with an AT&T license could get it and most people
did. Remember this is the system that BBN (Gurwitz) did the original IP/TCP support (sans
sockets - i.e. /dev/ip /dev/tcp ...). Your date of June '81 for the 4.1BSD release
seems late, but I'll accept it. 3BSD was 1979, and I thought 4BSD was a year later,
with 4.1BSD a few months after 4BSD (few people actually got 4BSD)
I am aware of that. “20 years of Berkely Unix” says November 1980 for 4BSD and June 1981
for 4.1BSD. From the SCCS log I’d say November 9th and July 1st respectively.
As far as I can tell CSRG integrated the BBN stack with (almost) 4.1 in April 1981 - most
of that is in a separate tree, with a few #ifdef’s in the main tree. From SCCS it seems
that implementation work on sockets started in Oct/Nov 1981 and 4.1a was released in April
or March 1982. From the partial coverage in SCCS I’d say there is no BBN code in 4.1a
anymore. The socket API in that release was different from what ended being in 4.1c and
4.2, and seems to have had some wider visibility in 2.9BSD and UniPlus System V - both of
which used the 4.1a API (and probably code base).
That said, Bill and Sam did the heavy lifting on
select(2) first in 4.1aBSD and there were some issues (again I have forgotten the details
-- I do remember, I was working on my thesis and I had a do a huge rewrite of the AP
kernel support to handle select(2) properly). I remember talking to Sam (arguing with him
most probably) about it one night before it was fully created. I want to say, he had
worked on something similar at the firm he was at (the firm name I now forget -- si-mumble
-- they were in Mt. View) before he joined CRSG. I don't remember now the issues I
had, but I do remember it was a bit of mess to support the way the AP hardware assumed it
could do DMA on the UBA[2]
[1] 4.1a/4.1b/4.1cBSD was officially internal to UCB
and some ARPA-sites, although I would have expected someone like Dennis in 1127 to have
been sent it also, as wnj was in the process of leaving for Sun and he took them with him.
For instance, I would take 4.1c to Masscomp. The key is that these were not as widely
distributed as 4.1BSD. 4.2BSD would really accelerate BSD UNIX uptake, because of the
networking support but there was more than 2-3 years between 4.1BSD and 4.2BSD.
Yes, Summer 81 - Summer 82 must have been a vibrant year, with much interaction between
various Unix groups. By the way, dmr was on the 4.2BSD steering group.
[2] The AP's MMU/DMA interface at the time, was
causing me great hair, and that was likely to have been part of the reason I wanted some
help/changes in the KPI interface - which is actually funny, they eventually came with the
CMU Mach MMU changes of 4.4BSD (which was much more friendly to a
multiprocessor/coprocessor architecture). FWIW: I never got them when I was at UCB.
7:31 p.m.
On Mon, Mar 30, 2020 at 12:02 PM Paul Ruizendaal <pnr(a)planet.nl> wrote:
Comments in line:
Ok, that sounds right 6-7 months between them. That was the time of the
'FASTVAX' work and the fight with Stanford over if BSD or VMS was to be the
official Arpa OS. I remember Joy was incredibly prolific during that
time. From a user standpoint, 4BSD and 4.1BSD are really similar, but he
was hacking the kernel. It seemed like a new change came out all the
time. The CAD group (where I was) always wanted the fastest system, so
there was pressure on me to follow him, but I was trying to make progress
on the AP work (my thesis) so I was a little loath to take the churn.
On 30 Mar 2020, at 15:26, Clem Cole
<clemc(a)ccc.com> wrote:
Your date of June '81 for the 4.1BSD release
seems late, but I'll accept
it. 3BSD was 1979, and I thought 4BSD was a year later, with 4.1BSD a few
months after 4BSD (few people actually got 4BSD)
I am aware of that. “20 years of Berkely Unix” says November 1980 for 4BSD
and June 1981 for 4.1BSD. From the SCCS log I’d say November 9th and July
1st respectively.
As far as I can tell CSRG integrated the BBN stack with (almost) 4.1 in
April 1981 - most of that is in a separate tree, with a few #ifdef’s in the
main tree.
I'm pretty sure that was mostly Eric Cooper (ecc) IIRC. We had a tape
directly from Gurwitz running in the CAD lab in Cory Hall, running on Xerox
3M Ethernet and because we needed it between our three 780s systems. We had
ordered the 3C501's from 3COM and we were going to be the Beta for the
Interlan board - which I wrote the original driver and gave it to Sam. I'm
not sure when IngVAX arrived, Eric Schmidt had Ing70 downstairs (and that
was the ArpaNet connect via a VDH up the hill to the IMP at LBL. We had
been running BerkNET over serial lines. Ernie and Kim were the two 780s
over in Evan's (the big pile of 750s and the C30 IMP were still in the
future).
From SCCS it seems that implementation work on sockets
started in Oct/Nov
1981 and 4.1a was released in April or March 1982.
That sounds about right. It was a bunch of wnj marathon hacking sessions.
From the partial coverage in SCCS I’d say there is no
BBN code in 4.1a
anymore.
Mumble ... I sort of have a hard time with that. I knew and worked with
both of them in my day and have heard both sides of the story. Joy had
Gurwitz's code and was hacking it - that is just a fact. Claiming pure
authorship is a tad extreme. For instance, Andy VanDam told me that the
whole mbuf's stuff is something Rob originally wrote for a project for him
at Brown when he was an undergrad. Rob brought it BBN and he used it for
the OS-independent TCP (i.e. the HP3000 version) originally and it landed
in the UNIX version. Joy hacked on it, but that's a data structure
directly from Rob. On the other hand, Joy types open curly brace, close
curly brace, and hacks in code the fastest of anyone I ever knew (someone,
maybe Mary Ann Horton, once said he 'wrote code at 9600'). But if you look
at it, its sometimes not pretty. It works and its usually fast, which is
hard to argue with.
The socket API in that release was different from what
ended being in 4.1c
and 4.2,
Definitely. I thought other hands besides Joy started to mess with it
(like Sam)
and seems to have had some wider visibility in 2.9BSD
and UniPlus System V
- both of which used the 4.1a API (and probably code base).
Yeah, Keith was taking things was Sam at pretty hectic pace trying to keep
the 11's running. I would guess that took more of a 'good enough'
attitude, they got the TCP stack in the limited address space and did not
want to try much more
Yes, Summer 81 - Summer 82 must have been a vibrant year, with much
interaction between various Unix groups.
It was a fun time to be apart of it all.
By the way, dmr was on the 4.2BSD steering group.
Yes, I know. That's how I first got to know him.
31 Mar
31 Mar
2:44 a.m.
[...] the whole mbuf's stuff is something [...]
I would prefer not to go into the BSD code history at this point in time, as I’d like to
stay focused on what was going on at Bell Labs between 7th and 8th edition, as far as
networking is concerned. Of course it was all interconnecting, and the topic of
mbuf's is a nice arc to take us ultimately back too streams.
Looking at network buffering in Unix, I think the story begins in 1973 with Spider. In
Spider, the network driver was a character mode device, but used disk block buffering for
storage. This worked well, because the Spider interface sent/received data through DMA, in
cells of 256 bytes and at speeds comparable to contemporary disks:
https://minnie.tuhs.org/cgi-bin/utree.pl?file=V4/nsys/dmr/tdir/tiu.c.
In 1975 the UoI guys doing Arpa Unix had to interface to an IMP, which was much slower
than a disk but much faster than a terminal. They concluded that neither clists not disk
buffers were a good match and Steve Bunch came up with ’netbuf’s, which were 64 byte
blocks (carved out of a pool of disk blocks). Later (1978?) this code was modified to
optionally use separate buffer heads and buffer data, as part of the UoC buffer mod’s (to
alleviate the pressure on a limited address space):
https://minnie.tuhs.org/cgi-bin/utree.pl?file=SRI-NOSC/h/net/netbuf.h
In 1979 the first Datakit software was done (Chesson), where I don’t have much
information. My hypothesis is that this used clists, but I’m still trying to find a copy
of his paper about this implementation. Maybe the driver used a precursor to a streams
like data structure. Maybe this is also where MPX files fit in. Unfortunately, currently I
just don’t know.
In 1980 new TCP software for the VAX is being written, and again it needs to find a data
buffer approach. This is ‘mbuf’s, which are similar to the earlier ’netbuf’s. The main
changes are that the size is increased to 128 bytes and that a start offset is added,
which makes it possible to remove headers without copying the buffer. The ‘memberof’ field
is replaced by a second linkage pointer. The UoC idea to separate the header and data is
dropped again. They (mbufs) may have other origins as well, but as far as Unix networking
is concerned they are an evolution of the Arpa Unix ‘netbuf’s:
https://minnie.tuhs.org/cgi-bin/utree.pl?file=BBN-Vax-TCP/bbnnet/mbuf.h
Then we get to (late) 1981. On the Labs side streams are created, in my current
understanding mainly to support networking and in particular Datakit. The difference with
Spider is that the Datakit interface was not DMA based and that the data arrives in 9 bit
envelopes instead of 8 bit bytes. The 9th bit distinguishes between data bytes and control
bytes. Again neither clists nor disk buffers are a good match for buffering and a new
solution is created. Looking at buffers, these consist of a header and a separate
datablock. The datablock can be of various sizes: 4, 16, 64 and 1024 bytes - a match with
the varying nature of network traffic. Each block has a type (a.o. DATA and CONTROL) which
has various uses, but certainly gives a home to the 9th Datakit bit where needed:
https://github.com/Alhadis/Research-Unix-v8/blob/master/v8/usr/include/sys/…
https://github.com/Alhadis/Research-Unix-v8/blob/master/v8/usr/sys/dev/stre…
Over on the west coast the mbuf structure is extended with a new field, “type” at the same
time, and it transitions from being a pure network data buffer to a more generic buffer.
It remains at a fixed size of 128 bytes with inline data; the option to store data in a
separate block of varying size will only come back late in 4.3BSD. For the 4.2 version:
https://minnie.tuhs.org/cgi-bin/utree.pl?file=4.2BSD/usr/src/sys/h/mbuf.h
When viewed at this code level it would seem to me that by various routes everybody had
arrived at more or less the same the point in network buffering approach by 1981. That has
some interesting implications. At this level, is there really a difference between a pipe
implemented as cross connected streams and a pipe implemented as cross connected sockets?
Is an unbound socket not about the same as an unconnected stream head?
But before getting into such discussions, I’d really like to get the development story
from 7th to 8th edition as clear and precise as possible after 40 years.
On 30 Mar 2020, at 21:31, Clem Cole
<clemc(a)ccc.com> wrote:
On Mon, Mar 30, 2020 at 12:02 PM Paul Ruizendaal <pnr(a)planet.nl> wrote:
Comments in line:
On 30 Mar 2020, at 15:26, Clem Cole
<clemc(a)ccc.com> wrote:
Your date of June '81 for the 4.1BSD release
seems late, but I'll accept it. 3BSD was 1979, and I thought 4BSD was a year later,
with 4.1BSD a few months after 4BSD (few people actually got 4BSD)
I am aware of that. “20 years of Berkely Unix” says November 1980 for 4BSD and June 1981
for 4.1BSD. From the SCCS log I’d say November 9th and July 1st respectively.
Ok, that sounds right 6-7 months between them. That was the time of the
'FASTVAX' work and the fight with Stanford over if BSD or VMS was to be the
official Arpa OS. I remember Joy was incredibly prolific during that time. From a user
standpoint, 4BSD and 4.1BSD are really similar, but he was hacking the kernel. It seemed
like a new change came out all the time. The CAD group (where I was) always wanted the
fastest system, so there was pressure on me to follow him, but I was trying to make
progress on the AP work (my thesis) so I was a little loath to take the churn.
As far as I can tell CSRG integrated the BBN stack with (almost) 4.1 in April 1981 - most
of that is in a separate tree, with a few #ifdef’s in the main tree.
I'm pretty sure that was mostly Eric Cooper (ecc) IIRC. We had a tape directly from
Gurwitz running in the CAD lab in Cory Hall, running on Xerox 3M Ethernet and because we
needed it between our three 780s systems. We had ordered the 3C501's from 3COM and we
were going to be the Beta for the Interlan board - which I wrote the original driver and
gave it to Sam. I'm not sure when IngVAX arrived, Eric Schmidt had Ing70 downstairs
(and that was the ArpaNet connect via a VDH up the hill to the IMP at LBL. We had been
running BerkNET over serial lines. Ernie and Kim were the two 780s over in Evan's
(the big pile of 750s and the C30 IMP were still in the future).
From SCCS it seems that implementation work on sockets started in Oct/Nov 1981 and 4.1a
was released in April or March 1982.
That sounds about right. It was a bunch of wnj marathon hacking sessions.
From the partial coverage in SCCS I’d say there is no BBN code in 4.1a anymore.
Mumble ... I sort of have a hard time with that. I knew and worked with both of them
in my day and have heard both sides of the story. Joy had Gurwitz's code and was
hacking it - that is just a fact. Claiming pure authorship is a tad extreme. For
instance, Andy VanDam told me that the whole mbuf's stuff is something Rob originally
wrote for a project for him at Brown when he was an undergrad. Rob brought it BBN and he
used it for the OS-independent TCP (i.e. the HP3000 version) originally and it landed in
the UNIX version. Joy hacked on it, but that's a data structure directly from Rob.
On the other hand, Joy types open curly brace, close curly brace, and hacks in code the
fastest of anyone I ever knew (someone, maybe Mary Ann Horton, once said he 'wrote
code at 9600'). But if you look at it, its sometimes not pretty. It works and its
usually fast, which is hard to argue with.
The socket API in that release was different from what ended being in 4.1c and 4.2,
Definitely. I thought other hands besides Joy started to mess with it (like Sam)
and seems to have had some wider visibility in 2.9BSD and UniPlus System V - both of
which used the 4.1a API (and probably code base).
Yeah, Keith was taking things was Sam at pretty hectic pace trying to keep the 11's
running. I would guess that took more of a 'good enough' attitude, they got
the TCP stack in the limited address space and did not want to try much more
Yes, Summer 81 - Summer 82 must have been a vibrant year, with much interaction between
various Unix groups.
It was a fun time to be apart of it all.
By the way, dmr was on the 4.2BSD steering group.
Yes, I know. That's how I first got to know him. 
7:57 p.m.
I would be interested too. That was a seminal time with the
proliferation of graphical programs, such as jim, pads/pi,
proof, the sophisticated menus of 'mhit.c' etc. It is curious
that little if any of that code made the transition to Plan9
(apart from sam, nee jim).
On Sun, Apr 5, 2020 at 12:19 AM emanuel stiebler <emu(a)e-bbes.com> wrote:
On 2020-03-30 05:06, Rob Pike wrote:
I've looked through my notes and
unfortunately there's very little
about this as the notes are mostly about graphics and physics.
I wouldn't mind seeing the note baout graphics, even if not on topic for
this group ...
9:32 p.m.
Sam wasn't nee jim. Jim was a toy, sam is a serious editor.
My notebooks are hundreds of pages of figuring stuff out. Perhaps
valuable information to historians, but not easily compressed for this
forum.
-rob
On Sun, Apr 5, 2020 at 5:57 AM Noel Hunt <noel.hunt(a)gmail.com> wrote:
I would be interested too. That was a seminal time with the
proliferation of graphical programs, such as jim, pads/pi,
proof, the sophisticated menus of 'mhit.c' etc. It is curious
that little if any of that code made the transition to Plan9
(apart from sam, nee jim).
On Sun, Apr 5, 2020 at 12:19 AM emanuel stiebler <emu(a)e-bbes.com> wrote:
>
> On 2020-03-30 05:06, Rob Pike wrote:
> > I've looked through my notes and unfortunately there's very little
> > about this as the notes are mostly about graphics and physics.
>
> I wouldn't mind seeing the note baout graphics, even if not on topic for
> this group ...
10:39 p.m.
Sam wasn't nee jim. Jim was a toy, sam is a
serious editor.
I don't doubt that, as a long-time user of sam (and jim
initially, 30 or so years ago), but it is interesting to
see the gradual sophistication of, say, the frame library
presumably in response to changes in hardware, and so on.
5 Apr
5 Apr
1:38 a.m.
On Sun, Apr 05, 2020 at 07:32:22AM +1000, Rob Pike wrote:
Sam wasn't nee jim. Jim was a toy, sam is a
serious editor.
My notebooks are hundreds of pages of figuring stuff out. Perhaps
valuable information to historians, but not easily compressed for this
forum.
I think this info would be more than interesting.
Maybe if and when you have the time,
you could compress this (at least for your blog)?
1:17 p.m.
On 2020-04-04 17:32, Rob Pike wrote:
Sam wasn't nee jim. Jim was a toy, sam is a
serious editor.
My notebooks are hundreds of pages of figuring stuff out. Perhaps
valuable information to historians, but not easily compressed for this
forum.
If you need anybody with a scanner an patience, please tell me ;-)
-rob
On Sun, Apr 5, 2020 at 5:57 AM Noel Hunt <noel.hunt(a)gmail.com> wrote:
>
> I would be interested too. That was a seminal time with the
> proliferation of graphical programs, such as jim, pads/pi,
> proof, the sophisticated menus of 'mhit.c' etc. It is curious
> that little if any of that code made the transition to Plan9
> (apart from sam, nee jim).
>
>
> On Sun, Apr 5, 2020 at 12:19 AM emanuel stiebler <emu(a)e-bbes.com> wrote:
>>
>> On 2020-03-30 05:06, Rob Pike wrote:
>>> I've looked through my notes and unfortunately there's very little
>>> about this as the notes are mostly about graphics and physics.
>>
>> I wouldn't mind seeing the note baout graphics, even if not on topic for
>> this group ...
8 Apr
8 Apr
6:50 p.m.
As was suggested on the list, I’ve reached out to Peter Weinberger to better understand
the time line of the File System Switch and the 8th edition network file system. He has
been very helpful, but the one line summary is that it is unfortunately too long ago to
remember specific details with any certainty. In general Peter remembers that he was
concerned that the project was too big for one person to do, and hence always looked for
design choices that would leave the work scope manageable.
Time line.
Since my last post on this subject I have found that the ACM conference talk of March 1985
was also held 9 months earlier at a Usenix conference - leaving a time slot between end of
1981 and summer 1984. Peter vaguely remembers that the essential ideas were done
"before 1983”. It would stand to reason that 1983 was spent on getting corner cases
of the network file system right, but all this is no more than plausible conjecture.
File system switch (FSS).
The guiding thought for the FSS was to extend the philosophy of ‘everything is a file’ to
new areas, also other than network files. Early implementations already included a simple,
read-only ‘/proc’ file system for example. I asked if any experiments had been done with
virtualising ‘/dev', but Peter could not recall any such work.
I personally find that the FSS has an elegance that fits with other parts of Research Unix
and asked Peter about its origins. He does not recall any special "a-ha” moments, but
does recall that the way it was done just felt natural to him. Other options would have
included to do the switch at the sys call level (which felt too complicated) or at the
block device level (which felt too limited).
I also asked about how his reworking of ’namei’ and centralising all namespace operations
in that function came about (in my view it is key to a concise switch). Here, too, it is
too long ago to remember any specifics, but Peter comments that he never liked to write
much code and that spending time on finding ways to make the amount of coding as small and
straightforward as possible would have been in character.
Eighth edition network file system
Once the FSS exists, a simplistic network file system is not hard - just do RPC to a
remote server. Peter chose to do a user level file server in order to keep the work scope
and complexity down to manageable levels. As highlighted in the ACM paper, the devil is in
the detail of replicating all the semantics of normal local disk files. Cases like a file
being kept alive if a process still has it open, the complexities of cross-mounted network
files (let alone recursively mounted), handling failed connections, etc. were hard to sort
out and get right.
On 29 Mar 2020, at 20:12, Paul Ruizendaal
<pnr(a)planet.nl> wrote:
On 29 Mar 2020, at 16:04, arnold(a)skeeve.com wrote:
Paul Ruizendaal <pnr(a)planet.nl> wrote:
I have that paper (“The Unix 8th Edition Network File System”), it was presented at a
March 1985 ACM conference. However, there are indications that the roots of the file
system switch existed earlier, possibly much earlier.
I think Doug McIlroy once described 1973 as a pivotal year for Unix, with many concepts
devised that would blossom in the following 3-5 years. I’m increasingly tempted to think
that Summer ’81 - Summer ’82 was a similarly pivotal year.
Related is the question when the "file
system switch" was added. It must
have been later than 1981 and before 1985, but I have not been able to
pinpoint it further.
IIRC there was a "paper" (only an abstract) on the file system
switch published in a USENIX conference proceedings. That woud help
trace it down. Peter Weinberger, who did it, is at Google; you
could ask him
directly, as well.
That is a good idea. If someone has the email address I’d appreciate an off list
message.
Paul 
6:58 p.m.
As was suggested on the list, I’ve reached out to Peter
Weinberger to better
understand the time line of the File System Switch and the 8th edition network
f>ile system.
'Introduced with System V Release 3.0, the File System Switch (FSS) architecture
introduced a framework under which multiple different filesystem types could coexist in
parallel.'
https://www.oreilly.com/library/view/unix-filesystems-evolution/97804714567…
8:13 p.m.
On Apr 8, 2020, at 8:58 PM, Thomas Paulsen <thomas.paulsen(a)firemail.de> wrote:
Thanks for that link!
The SysV R3 source floats around on the web. Its FSS is very different from what is in 8th
edition.
In 8th edition the switch has 11 entries (i.e. a file system is an object with 11 virtual
methods).
https://github.com/Alhadis/Research-Unix-v8/blob/master/v8/usr/sys/h/conf.h
I have never really studied R3 but at quick inspection the FSS there has 27 (!) entries
and seems to be more a sys call switch.
In 10th edition it is still 11 entries, although some refactoring has taken place. Also
later work from Research keeps it concise: the 9P protocol from Plan 9 has 14 messages.
As was suggested on the list, I’ve reached out to
Peter Weinberger to better
understand the time line of the File System Switch and the 8th edition network
f>ile system.
'Introduced with System V Release 3.0, the File System Switch (FSS) architecture
introduced a framework under which multiple different filesystem types could coexist in
parallel.'
https://www.oreilly.com/library/view/unix-filesystems-evolution/97804714567…
<https://www.oreilly.com/library/view/unix-filesystems-evolution/9780471456759/chap07-sec003.html>
1472
days inactive
1482
days old
22 comments
10 participants
participants (10)
-
aksr -
arnold＠skeeve.com -
Clem Cole -
emanuel stiebler -
Jon Forrest -
Noel Hunt -
Paul Ruizendaal -
reed＠reedmedia.net -
Rob Pike -
Thomas Paulsen