TUHS January 2017

tuhs@tuhs.org

77 participants
95 discussions

by jnc＠mercury.lcs.mit.edu

> From: Johnny Billquist <bqt(a)softjar.se> > And RFC791 is dated September 1981. Yes, but it had pretty much only editorial changes from RFC-760, dated January 1980 (almost two years before), and from a number of IEN's dated even earlier than that (which I'm too lazy to paw through). > So I have this problem with people who say that they implemented TC/IP > in 1978 for some reason. If you look at IEN-44, June 1978 (issued shortly after the fateful June 15-16 meeting, where the awful 32-bit address decision was taken), you will see that the packet format as of that date was pretty much what we have today (the format of addresses kept changing for many years, but I'll put that aside for now). > Especially if they say ... it was working well in heterogeneous > networks. TCP/IP didn't work "well" for a long time after 1981 - until we got the congestion control stuff worked out in the late 80's. And IIRC the routing/ addressing stuff took even longer. > I don't think it's correct to say that it was TCP/IP, as we know it > today. Why not? A box implementing the June '78 spec would probably talk to a current one (as long as suitable addresses were used on each end). > It was either some other protocol (like NCP) or some other version of > IP, which was not even published as an RFC. Nope. And don't put too much weight on the RFC part - TCP/IP stuff didn't start getting published as RFC's until it was _done_ (i.e. ready for the ARPANet to convert from NCP to TCP/IP - which happened January 1, 1983). All work prior to TCP/IP being declared 'done' is documented in IEN's (very similar documents to RFC's, distributed by the exact same person - Jon Postel). Noel

7 years, 3 months

Re: [TUHS] History of select(2)

by Johnny Billquist

On 2017-01-13 18:57, Paul Ruizendaal <pnr(a)planet.nl> wrote: > > On 12 Jan 2017, at 4:54 , Clem Cole wrote: > >> The point is that while I have no memory of capac(), but I can confirm that I definitely programmed with the empty() system call and Rand ports on a v6 based kernel in the mid-1970s and that it was definitely at places besides Rand themselves. > Thank you for confirming that. If anybody knows of surviving source for these extensions I'd love to hear about it. Although the description in the implementation report is clear enough to recreate it (it would seem to be one file similar to pipe.c and a pseudo device driver similar in size to mem.c), original code is better. It is also possible that the code in pipe.c was modified to drive both pipes and ports -- there would have been a lot of similarity between the two, and kernel space was at a premium. > >> [...] confirming something I have been saying for few years and some people have had a hard time believing. The specifications for what would become IP and TCP were kicking around the ARPAnet in the late 1970s. > My understanding is that all RFC's and IEN's were available to all legit users of the Arpanet. By 1979 there were 90 nodes (IMP's) and about 200 hosts connected. I don't get the impression that stuff was always easy to find, with Postel making a few posts about putting together "protocol information binders". Apparently nobody had the idea to put all RFC's in a directory and give FTP access to it. They were, and still are. And I suspect Clem is thinking of me, as I constantly question his memory on this subject. The problem is that all the RFCs are available, and they are later than this. The ARPAnet existed in 1979, but it was not using TCP/IP. If you look at the early drafts of TCP/IP, from around 1980-1981, you will also see that there are significant differences compared to the TCP/IP we know today. There was no ICMP, for example. Error handling and passing around looked different. IMPs did not talk IP, just for the record. RFC760 defines IPv4, and is dated January 1980. It refers to some previous documents that describe IP, but they are not RFCs. Also, if you look at RFC760, you will see that errors were supposed to be handled through options in the packet header, and that IP addresses, while 32 bits, were just split into 8 bits for network number, and 24 bits for host. There were obviously still some work needed before we got to what people think on IPv4 today. Anyone implementing RFC760 would probably not work at all with an IPv4 implementation that exist today. > I am not sure how available this stuff was outside the Arpanet community. I think I should put a question out about this, over on the internet history mailing list. > > As an aside: IMHO, conceptually the difference between NCP and TCP wasn't all that big. In my current understanding the big difference was that NCP assumes in-order, reliable delivery of packets (as was the case between IMP's) and that TCP allows for unreliable links. Otherwise, the connection build-up and tear-down and the flow control were similar. See for instance RFC54 and RFC55 from 1970. My point is: yes, these concepts were kicking around for over a decade in academia before BSD. Not sure if BSD is a good reference point. Much stuff was not actually done on Unix systems at all, if you start reading machine lists in the early RFCs. Unix had this UUCP thingy, that they liked. ;-) BSD and networking research came more to the front doing all the refinements over the years. Anyway, yes, for sure TCP did not come out of the void. It was based on earlier work. But there are some significant differences between TCP/IP and NCP, which is why you had the big switch day. Johnny -- Johnny Billquist || "I'm on a bus || on a psychedelic trip email: bqt(a)softjar.se || Reading murder books pdp is alive! || tryin' to stay hip" - B. Idol

7 years, 3 months

Re: [TUHS] 2.11BSD cross compiling update

by Nick Downing

I see, no I had not realized that code is still in use, I would have thought it had been replaced by a whole lot of POSIX bloat. Admittedly the 2.11BSD ctime/asctime/localtime/timezone stuff is simplistic and doesn't address complicated cases but it's good enough. However I have to resist the temptation to improve or update stuff in 2.11BSD, I went down that path many times (with the Makefiles project for instance) and because everything is interdependent you always introduce more problems and get deeper and deeper enmeshed. In order to stay in control I only fix essentials and apply a rule of minimal change, period. This applies until I have a baseline that builds exactly the same binary system image as the native build. Then I might proactively improve parts of the system but I will not do it reactively if you follow. As I see it the zic behaviour is not a bug since time_t is 32 bits on 2.11BSD and has no unreasonable values, and localtime() is BSD not POSIX compliant and is not allowed to return NULL. cheers, Nick On 14/01/2017 6:53 AM, "Random832" <random832(a)fastmail.com> wrote: On Fri, Jan 13, 2017, at 12:57, Nick Downing wrote: > I then ended up doing a fair bit of re-engineering, how this came > about was that I had to port the timezone compiler (zic) to run on the > Linux cross compilation host, since the goal is eventually to build a > SIMH-bootable disk (filesystem) with everything on it. This was a bit > involved, it crashed initially and it turned out it was doing > localtime() on really small and large values to try to figure out the > range of years the system could handle. On the Linux system this > returns NULL for unreasonable time_t values which POSIX allows it to > do. Hence the crash. It wasn't too obvious how to port this code. (But > whatever I did, it had to produce the exact same timezone files as a > native build). You know that the timezone file format that it uses is still in use today, right? There's extra data at the end in modern ones for 64-bit data, but the format itself is cross-platform, with defined field widths and big-endian byte order. What do you get when you compare the native built timezone files with one from your linux host's own zic? It *should* only differ by the version number in the header [first five bytes "TZif2" vs "TZif"] and the 64-bit section, if you're giving it the same input files. And I bet you could take the current version of the code from IANA and, if it matters to you, remove the parts that output the 64-bit data. If nothing else, looking at the modern code and the version in 2.11BSD side-by-side will let you backport bug fixes. (Note: Technically, the version present in most Linux systems is a fork maintained with glibc rather than the main version of the code from IANA)

7 years, 3 months

2.11BSD cross compiling update

by Nick Downing

So I got a fair bit further advanced on my 2.11BSD cross compiler project, at the moment it can make a respectable unix tree (/bin /usr etc) with a unix kernel and most software in it. I unpacked the resulting tarball and chrooted into it on a running SIMH system and it worked OK, was a bit rough around the edges (missing a few necessary files in /usr/share and that kind of thing) but did not crash. I haven't tested the kernel recently but last time I tested it, it booted, and the checksys output is OK. I then ended up doing a fair bit of re-engineering, how this came about was that I had to port the timezone compiler (zic) to run on the Linux cross compilation host, since the goal is eventually to build a SIMH-bootable disk (filesystem) with everything on it. This was a bit involved, it crashed initially and it turned out it was doing localtime() on really small and large values to try to figure out the range of years the system could handle. On the Linux system this returns NULL for unreasonable time_t values which POSIX allows it to do. Hence the crash. It wasn't too obvious how to port this code. (But whatever I did, it had to produce the exact same timezone files as a native build). So what I ended up doing was to port a tiny subset of 2.11BSD libc to Linux, including its types. I copied the ctime.c module and prefixed everything with "cross_" so there was "cross_time_t" and so forth, and "#include <time.h>" became "#include <cross/time.h>", in turn this depends on "#include <cross/sys/types.h>" and so on. That way, the original logic worked unchanged. I decided to also redo the cross compilation tools (as, cc, ld, nm, ranlib and so on) using the same approach, since it was conceptually elegant. This involved making e.g. "cross_off_t" and "struct cross_exec" available by "#include <cross/a.out.h>", and obviously the scheme extends to whatever libc functions we want to use. In particular we use floating point, and I plan to make a "cross_atof()" for the C compiler's PDP-11-formatted floating-point constant handling, etc. (This side of things, like the cross tools, was working, but was not terribly elegant before). So then I got to thinking, actually this is an incredibly powerful approach. Instead of just going at it piecemeal, would it not be easier just to port the entire thing across? To give an example what I mean, the linker contains code like this: if (nund==0) printf("Undefined:\n"); nund++; printf("%.*s\n", NNAMESIZE, sp->n_name); It is printing n_name from a fixed-size char array, so to save the cost of doing a strncpy they have used that "%.*s" syntax which tells printf not to go past the end of the char array. But this isn't supported in Linux. I keep noticing little problems like this (actually I switched off "-Wformat" which was possibly a bad idea). So with my latest plan this will actually run the 2.11BSD printf() instead of the Linux printf(), and the 2.11BSD stdio (fixing various other breakage that occured because BUFSIZ isn't 512 on the Linux system), and so on. What I will do is, provide a low level syscalls module like cross_open(), cross_read(), cross_write() and so on, which just redirect the request into the normal Linux system calls, while adjusting for the fact that size_t is 16 bits and so forth. This will be really great. In case it sounds like this is over-engineering, well bear in mind that one knotty problem I hadn't yet tackled is the standalone utilities, for instance the 2.11BSD tape distribution contains a standalone "restor" program which is essentially a subset of the kernel, including its device drivers, packaged with the normal "restor" utility into one executable that can be loaded off the tape. It was quite important to me that I get this ported over to Linux, so that I can produce filesystems, etc, at the Linux level, all ready to boot when I attach them to SIMH. But it was going to be hugely challenging, since compiling any program that includes more than the most basic kernel headers would have caused loads of conflicts with Linux's kernel headers and system calls. So the new approach completely fixes all that. I did some experiments the last few days with a program that I created called "xify". What it does is to read a C file, and to every identifier it finds, including macro names, C-level identifiers, include file names, etc, it prepends the sequence "x_". The logic is a bit convoluted since it has to leave keywords alone and it has to translate types so that "unsigned int" becomes "x_unsigned_int" which I can define with a typedef, and so on. Ancient C constructs like "register i;" were rather problematic, but I have got a satisfactory prototype system now. I also decided to focus on 4.3BSD rather than 2.11BSD, since by this stage I know the internals and the build system extremely intimately, and I'm aware of quite a lot of inconsistencies which will be a lot of work to tidy up, basically things that had been hurriedly ported from 4.3BSD while trying not to change the corresponding 2.8~2.10BSD code too much. Also in the build system there are quite a few different ways of implementing "make depend" for example, and this annoys me, I did have some ambitious projects to tidy it all up but it's too difficult. So a fresh start is good, and I am satisfied with the 2.11BSD project up to this moment. So what will happen next is basically once I have "-lx_c" (the "cross" version of the 4.3BSD C library) including the "xified" versions of the kernel headers, then I will try to get the 4.3BSD kernel running on top of Linux, it will be a bit like User-Mode Linux. It will use simulated network devices like libpcap, or basically just whatever SIMH uses, since I can easily drop in the relevant SIMH code and then connect it up using the 4.3BSD kernel's devtab. The standalone utilities like "restor" should then "just work". The cross toolchain should also "just work" apart from the floating point issue, since it was previously targeting the VAX which is little-endian, and the wordsize issues and the library issues are taken care of by "xifying". Very nice. The "xifying" stuff is in a new repository 43bsd.git at my bitbucket (user nick_d2). cheers, Nick

7 years, 3 months

History of select(2)

by Warren Toomey

Also, I came across this history of select(2) a while back: https://idea.popcount.org/2016-11-01-a-brief-history-of-select2/ Cheers, Warren

7 years, 3 months

Re: [TUHS] Early Internet work (Was: History of select(2))

by jnc＠mercury.lcs.mit.edu

> From: Paul Ruizendaal >> On 12 Jan 2017, at 4:54 , Clem Cole wrote: >> The specifications for what would become IP and TCP were kicking around >> ... in the late 1970s. The whole works actually started considerably earlier than that; the roots go back to 1972, with the formation of the International Packet Network Working Group - although that group went defunct before TCP/IP itself was developed under DARPA's lead. I don't recall the early history well, in detail - there's a long draft article by Ronda Hauben which goes into it in detail, and there's also "INWG and the Conception of the Internet: An Eyewitness Account" by Alexander McKenzie which covers it too. By 1977 the DARPA-led effort had produced several working prototype implementations, and TCP/IP (originally there was only TCP, without a separate data packet carriage layer) were up to version 3. > My understanding is that all RFC's and IEN's were available to all legit > users of the Arpanet. Yes and no. The earliest distribution mechanism (for the initial NCP/ARPANet work) was hardcopy (you can't distribute things over the 'net before you have it working :-), and in fact until a recent effort to put them all online, not all RFC's were available in machine-readable form. (I think some IEN's still aren't.) So for many of them, if you wanted a copy, you had to have someone at ISI make a photocopy (although I think they stocked them early on) and physically mail it to you! > Apparently nobody had the idea to put all RFC's in a directory and give > FTP access to it. I honestly don't recall when that happened; it does seem obvious in retrospect! Most of us were creating document in online text systems, and it would have been trivial to make them available in machine-readable form. Old habits die hard, I guess... :-) > I think I should put a question out about this, over on the internet > history mailing list. Yes, good idea. > As an aside: IMHO, conceptually the difference between NCP and TCP > wasn't all that big. Depends. Yes, the service provided to the _clients_ was very similar (which can be seen in how similar the NCP and TCP versions of thing like TELNET, FTP, etc were), but internally, they are very different. > In my current understanding the big difference that was NCP assumes > in-order, reliable delivery of packets ... and that TCP allows for > unreliable links. Yes, that's pretty accurate (but it does mean that there are _a lot_ of differences internally - re-transmissions, etc). One other important difference is that there's no flow control in the underlying network (something that took years to understand and deal with properly). > yes, these concepts were kicking around for over a decade in academia > before BSD. TCP/IP was the product of a large, well-organized, DARPA-funded and -led effort which involved industry, academic and government players (the first two, for the most part, DARPA-funded). So I wouldn't really call it an 'academic' project. Noel

7 years, 3 months

VFS: confusion (and an end)

by Berny Goodheart

Thanks Warren for saving me to sort out the confusion. I am sorry I started it in the first place. On Tue, Jan 10, 2017 at 11:20 AM, Joerg Schilling <schily(a)schily.net <mailto:schily@schily.net>> wrote: > …. Note that > this list is very similar to that in the early part of his book on System V > internals. Having having just removed the dust off my old copy of TMGE, it is interesting that the list I wrote here is very similar to what I wrote back in 1993. Just goes to show, Alzheimer’s hasn’t got me yet ;)

7 years, 3 months

VFS: confusion (and an end)

by Warren Toomey

Ok, the story so far. Berny wrote: Here's the breakdown of SVR4 kernel lineage as I recall it. ... From SunOS: vnodes VFS Dan Cross wrote: > VFSSW <=== NO, this is from SunOS-4 Surely Berny meant the file system switch here, which could have come from early system V, but originated in research Unix (8th edition?). Joerg Schilling wrote: It is rather a part of the VFS interface that has first been completed with SunOS-3.0 in late 1985. And this is where the confusion starts. Does "It" refer to FSS or VFS? I've just looked through some sources. The file system switch was in SysVR3: uts/3b2/sys/mount.h: /* Flag bits passed to the mount system call */ #define MS_RDONLY 0x1 /* read only bit */ #define MS_FSS 0x2 /* FSS (4-argument) mount */ VFS was in SunOS 4.1.4: sys/sys/vfs.h: struct vfssw { char *vsw_name; /* type name string */ struct vfsops *vsw_ops; /* filesystem operations vector */ }; And VFS is in SysVR4: uts/i386/sys/vfs.h: typedef struct vfssw { char *vsw_name; /* type name string */ int (*vsw_init)(); /* init routine */ struct vfsops *vsw_vfsops; /* filesystem operations vector */ long vsw_flag; /* flags */ } vfssw_t; Interestingly, the "filesystem operations vector" comment also occurs in FreeBSD 5.3, NetBSD-5.0.2 and OpenBSD-4.6. Look for vector here: http://minnie.tuhs.org/cgi-bin/utree.pl?file=FreeBSD-5.3/sys/sys/mount.h http://minnie.tuhs.org/cgi-bin/utree.pl?file=NetBSD-5.0.2/sys/compat/sys/mo… http://minnie.tuhs.org/cgi-bin/utree.pl?file=OpenBSD-4.6/sys/sys/mount.h Larry wrote: System Vr3 had something called the file system switch which is what Berny is talking about. SunOS had virtual file system layer (VFS) and that would be one of things ported to SVr4. which is consistent with everybody else. So now that we have consistency, let's move on. Cheers, Warren

7 years, 3 months

the guy who brought up SVr4 on Sun machines

by Berny Goodheart

I have been trolling these many threads lately of interest. So thought I should chip in. "SVr4 was not based on SunOS, although it incorporated many of the best features of SunOS 4.x”. IMHO this statement is almost true (there were many great features from BSD too!). SunOS 5.0 was ported from SVR4 in early 1991 and released as Solaris 2.0 in 1992 for desktop only. Back in the late 80s, Sun and AT&T partnered development efforts so it’s no surprise that SunOS morphed into SVR4. Indeed it was Sun and AT&T who were the founding members of Unix International…with an aim to provide direction and unification of SVR4. I remember when I went to work for Sun (much later in 2003), and found that the code base was remarkably similar to the SVR4 code (if not exact in many areas). Here’s the breakdown of SVR4 kernel lineage as I recall it. I am pretty sure this is correct. But I am sure many of you will put me right if I am wrong ;) From BSD: TCP/IP C Shell Sockets Process groups and job Control Some signals FFS in UFS guise Multi groups/file ownership Some system calls COFF From SunOS: vnodes VFS VM mmap LWP and kernel threads /proc Dynamic linking extensions NFS RPC XDR From SVR3: .so libs revamped signals and trampoline code VFSSW RFS STREAMS and TLI IPC (Shared memory, Message queues, semaphores) Additional features in SVR4 from USL: new boot process. ksh real time extensions Service access facility Enhancements to STREAMS ELF

7 years, 3 months

Re: [TUHS] the guy who brought up SVr4 on Sun machines

by jnc＠mercury.lcs.mit.edu

> I wonder if >pdd ... was in any way any inspiration for /proc? That may have been a bit too cryptic. "pdd" ('process directory directory') was a top-level directory in the Multics filesystem which contained a directory for each process active in the system; each directory contained data (in segments - roughly, 'files', but Multics didn't have files because it was a single-level store system) associated with the process, such as its kernel- and user-mode (effectively - technically, ring-0 and ring-4) stacks, etc. So if a process was sitting in a system call, you could go into the right directory in >pdd and look at its kernel stack and see the sequence of procedure calls (with arguments) that had led it to the point where it blocked. Etc, etc. Noel

7 years, 3 months

← Newer
1
2
3
4
5
6
7
8
9
10
Older →

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

1996

1995

1994

1993

1992

1991

1990

TUHS January 2017