22 Sep
2017
22 Sep
'17
10:36 a.m.
Gentlemen,
Below some additional thoughts on the various observations posted
about this. Note that I was not a contemporary of these developments,
and I may stand corrected on some views.
I'm pretty sure the two main System V based
TCP/IP stacks were STREAMS
based: the Lachman one (which I ported to the ETA-10 and to SCO Unix)
and the Mentat one that was done for Sun. The socket API was sort of
bolted on top of the STREAMS stuff, you could get to the STREAMS stuff
directly (I think, it's been a long time).
Yes, that is my understanding too. I think it goes back to the two
roots of networking on Unix: the 1974 Spider network at Murray Hill and
the 1975 Arpanet implementation of the UoI.
It would seem that Spider chose to expose the network as a device, whereas
UoI chose to expose it as a kind of pipe. This seems to have continued in
derivative work (Datakit/streams/STREAMS and BBN/BSD sockets respectively).
When these systems were developed networking was mostly over serial lines,
and to use serial drivers was not illogical (i.e. streams->STREAMS). By 1980
fast local area networks were spreading, and the idea to see the network as
a serial device started to suck.
Much of the initial modification work that Joy did on the BBN code was to
make it perform on early ethernet -- it had been designed for 50 kbps arpanet
links. Some of his speed hacks (such as trailing headers) were later
discarded.
Interestingly, Spider was conceived as a fast network (1.5 Mbps); the local
network at Murray Hill operated at that speed, and things were designed to
work over long distance T1 connections as well. This integrated fast LAN/WAN
idea seems to have been abandoned in Datakit. I have a question out to Sandy
Fraser to ask about the origins of this, but have not yet received a reply.
The sockets stuff was something Joy created to compete
with the CMU Accent
networking system. [...] CMU was developing Accent on the Triple Drip
PascAlto (aka the Perq) and had a formal networking model that was very clean
and sexy. There were a lot of people interested in workstations, the Andrew
project (MIT is about to start Athena etc). So Bill creates the sockets
interface, and to show that UNIX could be just as modern as Accent.
I've always thought that the Joy/Leffler API was a gradual development of
the UoI/BBN API. The main conceptual change seem to have been support for
multiple network systems (selectable network stack, expansion
of the address space to 14 bytes).
I don't quite see the link to Accent and Wikipedia offers little help here
https://en.wikipedia.org/wiki/Accent_kernel
Could you elaborate on how Accent networking influenced Joy's sockets?
* There's no reason for
a separate listen() call (it takes a "backlog" argument but
in practice everyone defaults it and the kernel does strange
manipulations on it.)
Perhaps there is. The UoI/BBN API did not have a listen() call;
instead the open() call - if it was for a listening connection - blocked until
a connection occurred. The server process would then fork of a worker process
and re-issue the listening open() call for the next connection. This left a
time gap where the server would not be 'listening'.
The listen() call would create up to 'backlog' connection blocks in the
network code, so that this many clients could connect simultaneously
without user space intervention. Each accept() would hand over a (now
connected) connection block and add a fresh unconnected one to the backlog
list. I think this idea came from Sam Leffler, but perhaps he was inspired
by something else (Accent?, Chaos?)
Of course, this can be done with fewer system calls. The UoI/BBN system
used the open() call, with a pointer to a parameter data block as the 2nd
argument. Perhaps Joy/Leffler were of the opinion that parameter data
blocks were not very Unix-y, and hence spread it out over
socket()/connect()/bind()/listen() instead.
The UoI choice to overload the open() call and not create a new call
(analogous to the pipe() call) was entirely pragmatic: they felt this
was easier for keeping up with the updates coming out of Murray Hill
all the time.
In particular, I have often thought that it would have
been a better
and more consistent with the philosophy to have it implemented as
open("/dev/tcp") and so on.
I think this is perhaps an orthogonal topic: how does one map network names
to network addresses. The most ambitious was perhaps the "portal()" system
call contemplated by Joy, but soon abandoned. It may have been implemented
in the early 90's in BSD, but I'm not sure this was fully the same idea.
That said, making the the name mapping a user concern rather than a kernel
concern is indeed a missed opportunity.
Last and least, when feeling argumentative I would claim that connection
strings like "/dev/tcp/host:port" are simply parameter data blocks encoded
in a string :^)
This also knocks out the need for
SO_REUSEADDR, because the kernel can tell at the time of
the call that you are asking to be a server. Either someone
else already is (error) or you win (success).
Under TCP/IP I'm not sure you can. The protocol specifies that you must
wait for a certain period of time (120 sec, if memory serves my right)
before reusing an address/port combo, so that all in-flight packets have
disappeared from the network. Only if one is sure that this is not an
issue one can use SO_REUSEADDR.
Also, the profusion of system calls (send, recv,
sendmsg, recvmsg,
recvfrom) is quite unnecessary: at most, one needs the equivalent
of sendmsg/recvmsg.
Today that would indeed seem to make sense. Back in 1980 there seems
to have been a lot of confusion over message boundaries, even in
stream connections. My understanding is that originally send() and
recv() were intended to communicate a borderless stream, whereas
sendmsg() and recvmsg() were intended to communicate distinct
messages, even if transmitted over a stream protocol.
Paul
22 Sep
22 Sep
2:32 p.m.
New subject: Sockets and the true UNIX
Paul - you made a couple of comments and asked a few questions which I'll
take a shot at best I can having been there. Sam spent a few hours in my
apartment drinking beer and watching football, and sometimes doing laundry
much less and arguing networking stuff. (I had a washer and dryer in those
days and he had a vacuum cleaner which I borrowed from him). Before CSRG,
Sam had been working for a networking start up in the lower bay who's name
escapes me at the moment. IIRC they had been doing a terminal
concentrator of some type. Previous to UCB, I had been at Tek. Presotto
at DEC, Baden and George at Pr1me on the their 750.
Sundays were often a football and beer session, usually at Mike Carey and
my place.
On Fri, Sep 22, 2017 at 6:36 AM, Paul Ruizendaal <pnr(a)planet.nl> wrote:
Could you elaborate on how Accent networking influenced Joy's sockets?
...
The Accent system was definitely heavy on people's mind. The 1979 CMU
SPICE machine proposal had made quite a splash (I should scan it, I still
have an original copy). Bert Halstead and Gettys later told me the story
at MIT how they had a meeting when it came out and the whole tone was 'what
are we going to do what the CMU thing' (the result was Athena).
By the time I got Berkeley, most of the community was watching what was
going on. The issue in the end with Accent was that they did wrote it in
Pascal and it was not done with a UNIX API base (hence Mach would be C and
build into BSD). But Accent was the new cool system and UNIX was not
(yet) centered as the 'the' system in the DARPA community.
DARPA was still reeling from the move from the PDP-10, and a lot people
were not 100% happy with that (we have been discussing rms's hating Unix
and still pushing ITS as an example - he was hardly the only one). While
the VAX has the HW was officially the DARPA system, Stanford was pushing
for VMS if you remember. Some folks want different HW from the VAX too.
CMU & MIT were pushing DARPA for the idea of person workstations (Stanford
would join that too soon after an CMU EE would do his grad work at Stanford
- Andy B and build the "SUN" -- taking the ideas from the old CMU
distributed front-end project to a level no one ever imagined).
The point is that DARPA is kinda mixed up and the politics are not as clear
as it might seem 30 years later.
Rashid had presented the ports idea and whole Accent network model at a
DARPA PI meeting at Berkeley, I want to say in the Fall 1981-ish (date may
be off by a year or two). IIRC he gave the Berkeley systems seminar that
week too. All the UCB System people like me, had the paper and we were
abuzz about it. Mike Powell of Cray DemOS fame, was teaching the Grad OS
course and he was mixed up in the argument; as was Bart Miller (now at U
Wisc) who was one of his students then.
Mike has us all writing different papers and proposing different ideas. Who
specifically contributed what is hard to say at this point, although Joy
was the master craftsman and final author/implementor of what went into the
CRSG system. At lot of test stuff was done in private code or in other UCB
OS's work.
But Joy was absolutely trying to come up with something for UNIX that would
give a lot of the functionality of the Accent in more UNIX centric manner
because he felt he needed to keep the DARPA focused on UNIX as the system
of choice. Hence overlaying the file descriptor in the socket call while
Rashid in Mach, would leave ports as separate concept.
I
think this idea came from Sam Leffler, but perhaps he was inspired
by something else (Accent?, Chaos?)
Sam is a good friend and I give him credit for many things. Primarily as
making sure the BSD system worked. As I have always said about Joy, "he
types open curly brace, close curly brace and he patches faster than anyone
I have even known."
So, besides mopping up after Bill, the three things I know you can credit
to Sam are routed, rcp/rsh, and trailers in the headers. Besides that, he
was in on everything, but what was his, what was Bills, what was someone's
else's is hard.
As one of my friends who did his undergrad @ MIT who was also a UCB grad
student when I there put it (describing MIT's job control that Joy swiped
and added to BSD), "Bill recognized a lot of great ideas and then peed on
them to make them smell like UCB."
...
Last and least, when feeling argumentative I would claim that connection
strings like "/dev/tcp/host:port" are simply parameter data blocks encoded
in a string :^)
Exactly... Although the way that worked, was actually a side effect/bug in
name.
...
Back in 1980 there seems
to have been a lot of confusion over message boundaries
It really was before 1980. Those of us that lived networking in the 70s,
we were still trying to figure out what was the right way to do a lot of
things. By the early 80s we agreed we had to have it, but best known
methods were still off. We were still fighting the wars of SNA vs.
DECnet vs Wangnet vs MAP vs YourNet vs TelCo vs .... People had not yet
recognized Metcalfe's Law.
Which really is an example of where engineers forget the technology is
actually driven by economics. The 'winner' may not be the technology
that is the 'best' from 'theory.'
We forget that fact often on this list too I fear. I know it hurts, a
number of things I have been part in my career I 'know' were 'better'
than
what 'won' but it does not matter. They were not economical.
Clem
2:42 p.m.
New subject: Sockets and the true UNIX
On 9/22/17 10:32 AM, Clem Cole wrote:
(describing MIT's job control that Joy swiped
and added to BSD),
The original concepts might have come from MIT, but the BSD implementation
was done by Jim Kulp at IIASA before it was folded into 4.1 BSD.
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU chet(a)case.edu http://cnswww.cns.cwru.edu/~chet/
2:47 p.m.
New subject: Sockets and the true UNIX
On 9/22/17 10:42 AM, Chet Ramey wrote:
On 9/22/17 10:32 AM, Clem Cole wrote:
Though to be fair I think Kulp was an MIT guy who was inspired by ITS.
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU chet(a)case.edu http://cnswww.cns.cwru.edu/~chet/
(describing MIT's job control that Joy
swiped
and added to BSD),
The original concepts might have come from MIT, but the BSD implementation
was done by Jim Kulp at IIASA before it was folded into 4.1 BSD. 
2:52 p.m.
New subject: Sockets and the true UNIX
Chet Ramey wrote:
Yes, that's what he wrote to me when I asked him about it.
Clem Cole
wrote:
Though to be fair I think Kulp was an MIT guy who was inspired by ITS. (describing MIT's job control that Joy
swiped
and added to BSD),
The original concepts might have come from MIT, but the BSD
implementation
was done by Jim Kulp at IIASA before it was folded into 4.1 BSD. 
2:49 p.m.
New subject: Sockets and the true UNIX
On Fri, Sep 22, 2017 at 10:42:19AM -0400, Chet Ramey wrote:
On 9/22/17 10:32 AM, Clem Cole wrote:
Huh, I too thought Bill Joy did it. Did he integrate it and get credit
that way?
(describing MIT's job control that Joy
swiped
and added to BSD),
The original concepts might have come from MIT, but the BSD implementation
was done by Jim Kulp at IIASA before it was folded into 4.1 BSD.
2:57 p.m.
New subject: Sockets and the true UNIX
On 9/22/17 10:49 AM, Larry McVoy wrote:
On Fri, Sep 22, 2017 at 10:42:19AM -0400, Chet Ramey
wrote:
Kulp did the signal work in the kernel and added the code to the c shell
to use it (or had Eric Cooper, who was working for him as a student at
the time do fg/bg). He did his work on 3BSD and set the mods to Berkeley,
where Joy took them and integrated them into 4.1 BSD.
As far as I can tell, Kulp has always gotten credit for doing the initial
work, Joy for the `standard' BSD integration, and David Korn for doing
the second (non-csh) implementation.
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU chet(a)case.edu http://cnswww.cns.cwru.edu/~chet/
On 9/22/17 10:32 AM, Clem Cole wrote:
Huh, I too thought Bill Joy did it. Did he integrate it and get credit
that way? (describing MIT's job control that Joy
swiped
and added to BSD),
The original concepts might have come from MIT, but the BSD implementation
was done by Jim Kulp at IIASA before it was folded into 4.1 BSD. 
6:14 p.m.
New subject: Sockets and the true UNIX
Kulp did the signal work in the kernel and added the
code to the c shell
to use it (or had Eric Cooper, who was working for him as a student at
the time do fg/bg). He did his work on 3BSD and set the mods to Berkeley,
where Joy took them and integrated them into 4.1 BSD.
Kulp's name (and IIASA) appeared in several source files, I think;
I know I came across them.
Job control is a little overly complicated but there's only one
really big mistake in it, in my opinion, and that is that the
process group IDs are not allocated in the kernel. This is what
led to all the "session group leader" cruft in POSIX.
Job control ought to be independent of tty devices: each shell
pipeline is a "job", and the shell should group them by having the
kernel create an initial empty(ish -- the shell is a member of
both its original pgroup and this one it just created, until it
detaches from it) job, then pushing each process it forks into
that pgroup.
I'll use "fd" as a name here since I think that's one way to
return them (they're refcounted like file descriptors) but not
the only way, and it probably makes more sense to return a
pgroup ID directly and have a pgclose()...
// in a shell:
fd = new_pgroup(); // kernel allocates pgroup
for (all commands in a pipeline) {
pid = fork();
if (pid == 0) {
// child
pid = getpid();
setpgrp(pid, fd);
... all the other stuff we do ...
exec(childprocess)
}
if (pid < 0) ... clean up ...
// still in the shell - set up the rest of the pipe using
// the same pgroup
}
close(fd); // we're done with the pgroup, it goes away
// when the last pid we created above also exits
Note that this would mean that anything that creates login sessions
would have to have this same kind of code (create a pgroup, fork,
push child into pgroup, exec shell). But that's already true for
job control in general: init or getty needs to be job-control-aware,
and this makes their code simpler.
Chris
6:43 p.m.
New subject: Sockets and the true UNIX
On Fri, Sep 22, 2017 at 2:14 PM, Chris Torek <torek(a)torek.net> wrote:
Kulp's name (and IIASA) appeared in several
source files,
Which is exactly my point....As Larry said -- "*Huh, I too thought Bill
Joy did it. Did he integrate it and get credit that way?"*
Most people, even really smart ones that know a lot of the history like
Larry, think job control came from BSD. Bill recognized a good idea (and
I believe it came in on the PDP-11 version from IIASA and MIT BTW) and then
peed on it as it was made part of 4.1BSD.
To his credit, he was amazing at recognizing some great ideas and
integrating them; and some of them make the system much more usable and I
could not live without (I really would not want to type day-2-day to V6 or
V7 - which I did years ago and love it ->> then <<).
The core BSD system is the basic UNIX that the ROMs in my fingers and my
brain expect when I'm typing. Bill was the person that shepherded it and I
thank him. But it was a work of a lot people, but at UCB (and not just
CSRG) and outside.
7:08 p.m.
New subject: Sockets and the true UNIX
On 9/22/17 2:14 PM, Chris Torek wrote:
Job control ought to be independent of tty devices:
each shell
pipeline is a "job", and the shell should group them by having the
kernel create an initial empty(ish -- the shell is a member of
both its original pgroup and this one it just created, until it
detaches from it) job, then pushing each process it forks into
that pgroup.
In that sense, job control and process groups *are* independent of
the terminal device.
The complication arises when you have to multiplex one terminal
between several different jobs in an interactive shell. Setting
the terminal process group correctly -- the only way to accomplish
that -- is fragile and the consequences of setting it to the wrong
value are inconvenient, to say the least.
Chet
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU chet(a)case.edu http://cnswww.cns.cwru.edu/~chet/
8:57 p.m.
New subject: Sockets and the true UNIX
In that sense, job control and process groups *are*
independent of
the terminal device.
The complication arises when you have to multiplex one terminal
between several different jobs in an interactive shell. Setting
the terminal process group correctly -- the only way to accomplish
that -- is fragile and the consequences of setting it to the wrong
value are inconvenient, to say the least.
Sure, but here again you can have the shell hang on to the
(kernel allocated) pgroup, and any control operation amounts
to, e.g., "set tty pgroup to the one for my pgroupid X".
Since the shell is waiting for everything in the pipeline
to exit, the shell knows when to close its "pgid descriptor".
(If you abandon the pipeline entirely, you also abandon the
ability to attach it to a tty. There should be some way to
"rejoin" pgroups, which would let you attach them to new tty
sessions, which is something that is missing from the current
system. But that's another thing entirely.)
Chris
24 Sep
24 Sep
6:04 p.m.
New subject: Sockets and the true UNIX
On 9/22/17 4:57 PM, Chris Torek wrote:
That's not the problem this solves. The problem with allocating
process groups in the shell (and making them equal to the process id
of the pgrp leader) is that you end up with a race condition when
trying to add subsequent processes in a pipeline to that pgrp.
It forces you to artifically prevent the process group leader from
running or exiting until you're finished setting up the entire
pipeline.
The shell has to set the terminal process group to arbitrary process
groups, not just its own: that's how `fg' has to work. For that purpose,
tcsetpgrp is fine. You just have to make sure that you only call it
when the shell is in the foreground.
As for other processes, I'm not sure exactly what you mean by `control
operation', but you surely would want a background process to get stopped
if it tries to set the terminal attributes.
In that sense,
job control and process groups *are* independent of
the terminal device.
The complication arises when you have to multiplex one terminal
between several different jobs in an interactive shell. Setting
the terminal process group correctly -- the only way to accomplish
that -- is fragile and the consequences of setting it to the wrong
value are inconvenient, to say the least.
Sure, but here again you can have the shell hang on to the
(kernel allocated) pgroup, and any control operation amounts
to, e.g., "set tty pgroup to the one for my pgroupid X".
Since the shell is waiting for everything in the pipeline
to exit, the shell knows when to close its "pgid descriptor". (If you abandon the pipeline entirely, you also
abandon the
ability to attach it to a tty. There should be some way to
"rejoin" pgroups, which would let you attach them to new tty
sessions, which is something that is missing from the current
system. But that's another thing entirely.)
This is something that would be nice to have, but there are security
considerations that make it tricky. Not unsolvable, but tricky to get
right.
Chet
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU chet(a)case.edu http://cnswww.cns.cwru.edu/~chet/
22 Sep
22 Sep
6 p.m.
New subject: Sockets and the true UNIX
[I said]
> This also knocks out the need for SO_REUSEADDR ...
Under TCP/IP I'm not sure you can. The protocol
specifies that you must
wait for a certain period of time (120 sec, if memory serves my right)
before reusing an address/port combo, so that all in-flight packets have
disappeared from the network. Only if one is sure that this is not an
issue one can use SO_REUSEADDR.
That's independent of this issue. The problem is that when you
call bind(), the system does not know if you are doing it as
the first half of bind()/connect(), or as the first half of
bind()/listen( (followed by an accept() loop).
If you are going to do the latter, you are declaring that you want
to serve *new* SYN requests. This is when you (currently) must set
SO_REUSEADDR, because bind() checks your local address against the
universe of all local/remote pairs. But if the system knows that
your call (which, remember, isn't bind() anymore) is your
declaration of intent to be a server, the OS can just check
whether there's already someone declared as "the server".
If you intend to "serve" one specific foreign address (FTP):
bind_and_connect(fd, local, remote);
the test is: "does <local,remote> specify a unique address"
(and possibly "does <local> match what we already suggested
to you and reserved for you). If you intend to serve "*":
bind_and_connect(fd, local, NULL);
the test is: "is there already a <local,*>".
In the server case any existing 2MSL timeouts are irrelevant;
what matters is whether you are competing with another server.
(Note that libc will probably have various convenience wrappers
atop the system call, kind of like libc has system() atop
fork/exec. Most programs just want to create and bind_and_connect
a socket in one pass.)
Chris
2406
days inactive
2408
days old
12 comments
6 participants
participants (6)
-
Chet Ramey -
Chris Torek -
Clem Cole -
Larry McVoy -
Lars Brinkhoff -
Paul Ruizendaal