23 Jun
2020
23 Jun
'20
11:09 a.m.
When googling for File System Switch or Virtual File System most sources mention Sun NFS
and SysVr3 as the earliest implementations. Some sources mention 8th Edition.
I did a (short) search on FSS/VFS in earlier, non-Unix OS’s (Tenex, Multics, CTSS, etc.),
but none of those seem to have had a comparable concept.
Does anybody recall prior art (prior to 1984) in this area?
Paul
23 Jun
23 Jun
4:01 p.m.
I like to claim credit for creating the ChangeSet concept, the grouping
of a set of deltas across a number of files. But I wasn't the first.
Back when dejanews was a thing and you could search usenet posts in a
date range, if you searched before I started talking about ChangeSets,
you could find 6, count 'em, 6 hits. There was a really obscure system
called Aide De Camp, that had a similar concept.
But if you searched after I started talking about them you would see
millions of hits.
So I didn't invent the concept but I sure as heck made the world
understand the concept.
I suspect Sun could be in a similar position. The VFS concept is
pretty sweet so there might have been someone before Sun. I'll
long odds that if there was, it didn't gain traction until Sun
did it.
If there is nothing that predates it, then the inspiration was
almost certainly the device driver interface. One interface,
many devices. VFS is the same.
On Tue, Jun 23, 2020 at 11:09:57AM +0200, Paul Ruizendaal wrote:
When googling for File System Switch or Virtual File
System most sources mention Sun NFS and SysVr3 as the earliest implementations. Some
sources mention 8th Edition.
I did a (short) search on FSS/VFS in earlier, non-Unix OS???s (Tenex, Multics, CTSS,
etc.), but none of those seem to have had a comparable concept.
Does anybody recall prior art (prior to 1984) in this area?
Paul
--
---
Larry McVoy lm at mcvoy.com http://www.mcvoy.com/lm
5:12 p.m.
I agree with Larry with his observations. The only thing I would add is
that Popek and Walker had a file system in the UCLA Locus distributed
system for the 11/70 that was published in SIGOPS in late 1970s. In fact,
Gerry came up with the idea after a sabbatical @ PARC. But it was not
formalized like NFS as a separate (layered) FS, it was just part of the
basic Locus FS.
Peter Weinberger created the file system switch (FSS) for V8 and in the
UNIX world, it was first. Dave Arnovitz used it for RFS in System V,
while Perry Flynn and I used it for EFS, both of us after talking
to Peter. Rusty and team did the VFS layer Sun @ (Larry will have to tell
you who actually built it, I never knew). Rusty and I both published
papers in the '85 Summer USENIX (NFS and EFS were contemporaries -- the
difference is that Sun gave away NFS and Masscomp was not smart enough to
do that).
I talked to Steve Kleinman extensively about VFS at one point, and I'm
pretty sure he and the rest of the Sun guys had talked to the PARC folks
who after Gerry went back to UCLA started working on a DFS. The idea of
the state-less (idempotent) file system RPC that NFS used based on stuff
PARC did. But I'm not sure PARC had anything like the FSS or the VFS
layer. Peter, Dave, and I used a stateful scheme because we chose to have
full UNIX FS semantics, which NFS did not. In the end, early NFS was
notorious for putting 'holes' in the files because of the automatic seek in
every operation and errors not coming until close(2) time.
EFS used an RPC and a RUDP layer that Perry and Alan Atlas built, but it
was not nearly as flexible as what Sun built [which had a crude interface
generator], although until years later when Mike Leibensger built PIG (the
Paceline Interface Generator) NFS RPC was always a PITA and not much better
in practice than what we had at Masscomp.
In fact, the point of the EFS paper was it's all about the recovery when
there is a failure/error. If you read his paper, Rusty's point was who
cares if there is an error (I've always felt vindicated that while I lost
the war, over time everyone came to our way of thinking and now NFS
V4 looks a whole lot like EFS did).
Having a DFS as a feature was an incredible advance and proved we needed
something (and it needed to be standard in all systems). NFS really lead
the market with that advance, although it sort of took a few years to make
it really good. The fact is that others had the same idea before or at
least contemporary with it.
That said, and to give the NFS team* a huge amount of credit *(and great
applause), the VFS layer was better thought out than the FSS and in fact
made it possible to add a lot of different file systems into UNIX later.
FSS was much more ad hoc.
At LCC, when we built VPROC for TNC a few years later, we used some of the
same ideas from VFS and of course used VFS for the file system layer since
TNC had to have full POSIX semantics. (It's a shame VPROC never caught on
the way VFS did).
Clem
On Tue, Jun 23, 2020 at 10:02 AM Larry McVoy <lm(a)mcvoy.com> wrote:
I like to claim credit for creating the ChangeSet
concept, the grouping
of a set of deltas across a number of files. But I wasn't the first.
Back when dejanews was a thing and you could search usenet posts in a
date range, if you searched before I started talking about ChangeSets,
you could find 6, count 'em, 6 hits. There was a really obscure system
called Aide De Camp, that had a similar concept.
But if you searched after I started talking about them you would see
millions of hits.
So I didn't invent the concept but I sure as heck made the world
understand the concept.
I suspect Sun could be in a similar position. The VFS concept is
pretty sweet so there might have been someone before Sun. I'll
long odds that if there was, it didn't gain traction until Sun
did it.
If there is nothing that predates it, then the inspiration was
almost certainly the device driver interface. One interface,
many devices. VFS is the same.
On Tue, Jun 23, 2020 at 11:09:57AM +0200, Paul Ruizendaal wrote:
When googling for File System Switch or Virtual
File System most sources
mention Sun NFS and SysVr3 as the earliest
implementations. Some sources
mention 8th Edition.
I did a (short) search on FSS/VFS in earlier, non-Unix OS???s (Tenex,
Multics,
CTSS, etc.), but none of those seem to have had a comparable
concept.
Does anybody recall prior art (prior to 1984) in this area?
Paul
--
---
Larry McVoy lm at mcvoy.com
http://www.mcvoy.com/lm
5:55 p.m.
I recall listening to an interesting survey talk on DFS (pronounced Doofus)
variants at a USENIX ATC conference in the 80's. Someone here may be able
to recall the specifics...
-r
10:38 p.m.
For my taste, the various Unix file system switches that I've seen are too
firmly tied to the idea of blocks and disks and all that, making them less
flexible than they should have been. That's why the Plan 9 version is about
names and byte streams, to make it as general as possible.
That's one of the reasons the possibilities of the file system approach to
data has not reached its potential in Unix.
-rob
On Wed, Jun 24, 2020 at 1:14 AM Clem Cole <clemc(a)ccc.com> wrote:
I agree with Larry with his observations. The only
thing I would add is
that Popek and Walker had a file system in the UCLA Locus distributed
system for the 11/70 that was published in SIGOPS in late 1970s. In fact,
Gerry came up with the idea after a sabbatical @ PARC. But it was not
formalized like NFS as a separate (layered) FS, it was just part of the
basic Locus FS.
Peter Weinberger created the file system switch (FSS) for V8 and in the
UNIX world, it was first. Dave Arnovitz used it for RFS in System V,
while Perry Flynn and I used it for EFS, both of us after talking
to Peter. Rusty and team did the VFS layer Sun @ (Larry will have to tell
you who actually built it, I never knew). Rusty and I both published
papers in the '85 Summer USENIX (NFS and EFS were contemporaries -- the
difference is that Sun gave away NFS and Masscomp was not smart enough to
do that).
I talked to Steve Kleinman extensively about VFS at one point, and I'm
pretty sure he and the rest of the Sun guys had talked to the PARC folks
who after Gerry went back to UCLA started working on a DFS. The idea of
the state-less (idempotent) file system RPC that NFS used based on stuff
PARC did. But I'm not sure PARC had anything like the FSS or the VFS
layer. Peter, Dave, and I used a stateful scheme because we chose to have
full UNIX FS semantics, which NFS did not. In the end, early NFS was
notorious for putting 'holes' in the files because of the automatic seek in
every operation and errors not coming until close(2) time.
EFS used an RPC and a RUDP layer that Perry and Alan Atlas built, but it
was not nearly as flexible as what Sun built [which had a crude interface
generator], although until years later when Mike Leibensger built PIG (the
Paceline Interface Generator) NFS RPC was always a PITA and not much better
in practice than what we had at Masscomp.
In fact, the point of the EFS paper was it's all about the recovery when
there is a failure/error. If you read his paper, Rusty's point was who
cares if there is an error (I've always felt vindicated that while I lost
the war, over time everyone came to our way of thinking and now NFS
V4 looks a whole lot like EFS did).
Having a DFS as a feature was an incredible advance and proved we needed
something (and it needed to be standard in all systems). NFS really lead
the market with that advance, although it sort of took a few years to make
it really good. The fact is that others had the same idea before or at
least contemporary with it.
That said, and to give the NFS team* a huge amount of credit *(and great
applause), the VFS layer was better thought out than the FSS and in fact
made it possible to add a lot of different file systems into UNIX later.
FSS was much more ad hoc.
At LCC, when we built VPROC for TNC a few years later, we used some of the
same ideas from VFS and of course used VFS for the file system layer since
TNC had to have full POSIX semantics. (It's a shame VPROC never caught on
the way VFS did).
Clem
On Tue, Jun 23, 2020 at 10:02 AM Larry McVoy <lm(a)mcvoy.com> wrote:
I like to claim credit for creating the ChangeSet
concept, the grouping
of a set of deltas across a number of files. But I wasn't the first.
Back when dejanews was a thing and you could search usenet posts in a
date range, if you searched before I started talking about ChangeSets,
you could find 6, count 'em, 6 hits. There was a really obscure system
called Aide De Camp, that had a similar concept.
But if you searched after I started talking about them you would see
millions of hits.
So I didn't invent the concept but I sure as heck made the world
understand the concept.
I suspect Sun could be in a similar position. The VFS concept is
pretty sweet so there might have been someone before Sun. I'll
long odds that if there was, it didn't gain traction until Sun
did it.
If there is nothing that predates it, then the inspiration was
almost certainly the device driver interface. One interface,
many devices. VFS is the same.
On Tue, Jun 23, 2020 at 11:09:57AM +0200, Paul Ruizendaal wrote:
When googling for File System Switch or Virtual
File System most
sources mention Sun NFS and SysVr3 as the earliest
implementations. Some
sources mention 8th Edition.
I did a (short) search on FSS/VFS in earlier, non-Unix OS???s (Tenex,
Multics,
CTSS, etc.), but none of those seem to have had a comparable
concept.
Does anybody recall prior art (prior to 1984) in this area?
Paul
--
---
Larry McVoy lm at mcvoy.com
http://www.mcvoy.com/lm
10:57 p.m.
The SunOS one hid the block stuff pretty well. It had mmap().
Where it got hairy when I was working on it was block size 4k vs page
size 8k. And frags, I hate frags.
On Wed, Jun 24, 2020 at 06:38:53AM +1000, Rob Pike wrote:
For my taste, the various Unix file system switches
that I've seen are too
firmly tied to the idea of blocks and disks and all that, making them less
flexible than they should have been. That's why the Plan 9 version is about
names and byte streams, to make it as general as possible.
That's one of the reasons the possibilities of the file system approach to
data has not reached its potential in Unix.
-rob
On Wed, Jun 24, 2020 at 1:14 AM Clem Cole <clemc(a)ccc.com> wrote:
> I agree with Larry with his observations. The only thing I would add is
> that Popek and Walker had a file system in the UCLA Locus distributed
> system for the 11/70 that was published in SIGOPS in late 1970s. In fact,
> Gerry came up with the idea after a sabbatical @ PARC. But it was not
> formalized like NFS as a separate (layered) FS, it was just part of the
> basic Locus FS.
>
> Peter Weinberger created the file system switch (FSS) for V8 and in the
> UNIX world, it was first. Dave Arnovitz used it for RFS in System V,
> while Perry Flynn and I used it for EFS, both of us after talking
> to Peter. Rusty and team did the VFS layer Sun @ (Larry will have to tell
> you who actually built it, I never knew). Rusty and I both published
> papers in the '85 Summer USENIX (NFS and EFS were contemporaries -- the
> difference is that Sun gave away NFS and Masscomp was not smart enough to
> do that).
>
> I talked to Steve Kleinman extensively about VFS at one point, and I'm
> pretty sure he and the rest of the Sun guys had talked to the PARC folks
> who after Gerry went back to UCLA started working on a DFS. The idea of
> the state-less (idempotent) file system RPC that NFS used based on stuff
> PARC did. But I'm not sure PARC had anything like the FSS or the VFS
> layer. Peter, Dave, and I used a stateful scheme because we chose to have
> full UNIX FS semantics, which NFS did not. In the end, early NFS was
> notorious for putting 'holes' in the files because of the automatic seek
in
> every operation and errors not coming until close(2) time.
>
> EFS used an RPC and a RUDP layer that Perry and Alan Atlas built, but it
> was not nearly as flexible as what Sun built [which had a crude interface
> generator], although until years later when Mike Leibensger built PIG (the
> Paceline Interface Generator) NFS RPC was always a PITA and not much better
> in practice than what we had at Masscomp.
>
> In fact, the point of the EFS paper was it's all about the recovery when
> there is a failure/error. If you read his paper, Rusty's point was who
> cares if there is an error (I've always felt vindicated that while I lost
> the war, over time everyone came to our way of thinking and now NFS
> V4 looks a whole lot like EFS did).
>
> Having a DFS as a feature was an incredible advance and proved we needed
> something (and it needed to be standard in all systems). NFS really lead
> the market with that advance, although it sort of took a few years to make
> it really good. The fact is that others had the same idea before or at
> least contemporary with it.
>
> That said, and to give the NFS team* a huge amount of credit *(and great
> applause), the VFS layer was better thought out than the FSS and in fact
> made it possible to add a lot of different file systems into UNIX later.
> FSS was much more ad hoc.
>
> At LCC, when we built VPROC for TNC a few years later, we used some of the
> same ideas from VFS and of course used VFS for the file system layer since
> TNC had to have full POSIX semantics. (It's a shame VPROC never caught on
> the way VFS did).
>
> Clem
>
>
> On Tue, Jun 23, 2020 at 10:02 AM Larry McVoy <lm(a)mcvoy.com> wrote:
>
>> I like to claim credit for creating the ChangeSet concept, the grouping
>> of a set of deltas across a number of files. But I wasn't the first.
>> Back when dejanews was a thing and you could search usenet posts in a
>> date range, if you searched before I started talking about ChangeSets,
>> you could find 6, count 'em, 6 hits. There was a really obscure system
>> called Aide De Camp, that had a similar concept.
>>
>> But if you searched after I started talking about them you would see
>> millions of hits.
>>
>> So I didn't invent the concept but I sure as heck made the world
>> understand the concept.
>>
>> I suspect Sun could be in a similar position. The VFS concept is
>> pretty sweet so there might have been someone before Sun. I'll
>> long odds that if there was, it didn't gain traction until Sun
>> did it.
>>
>> If there is nothing that predates it, then the inspiration was
>> almost certainly the device driver interface. One interface,
>> many devices. VFS is the same.
>>
>> On Tue, Jun 23, 2020 at 11:09:57AM +0200, Paul Ruizendaal wrote:
>> > When googling for File System Switch or Virtual File System most
>> sources mention Sun NFS and SysVr3 as the earliest implementations. Some
>> sources mention 8th Edition.
>> >
>> > I did a (short) search on FSS/VFS in earlier, non-Unix OS???s (Tenex,
>> Multics, CTSS, etc.), but none of those seem to have had a comparable
>> concept.
>> >
>> > Does anybody recall prior art (prior to 1984) in this area?
>> >
>> > Paul
>>
>> --
>> ---
>> Larry McVoy lm at mcvoy.com
>> http://www.mcvoy.com/lm
>>
>
--
---
Larry McVoy lm at mcvoy.com http://www.mcvoy.com/lm
24 Jun
24 Jun
7:14 a.m.
Rob Pike <robpike(a)gmail.com> wrote:
For my taste, the various Unix file system switches
that I've seen are too
firmly tied to the idea of blocks and disks and all that, making them less
flexible than they should have been. That's why the Plan 9 version is about
names and byte streams, to make it as general as possible.
That's one of the reasons the possibilities of the file system approach to
data has not reached its potential in Unix.
-rob
Well, it's a progression, isn't it? As ideas are tried out, they are
used and then refined, and used and refined further. Multics did that
for its predecessors, Unix did that for Multics, and Plan 9 did that
for Unix. The key difference in Unix and Plan 9 from others being
the continual goal towards refinement through simplification.
Arnold
11:08 p.m.
On Tue, 23 Jun 2020, arnold(a)skeeve.com wrote:
Well, it's a progression, isn't it? As
ideas are tried out, they are
used and then refined, and used and refined further. Multics did that
for its predecessors, Unix did that for Multics, and Plan 9 did that for
Unix. The key difference in Unix and Plan 9 from others being the
continual goal towards refinement through simplification.
"Make things as simple as possible, but no simpler."
-- Dave
9:30 p.m.
Paul -- slow cache refill ... but I finally remembered some more of this
enough to reply...
I'm not sure if Larry remembers any of this and if he does, he might have
more details from the Sun side.
In the early/mid 90's, Dave Rosenthal was working on what become LOCKSS. I
think he was at still at Sun at that point, but he would eventually move to
Stanford fulltime, but he could have left Sun already - I've forgotten.
The UCLA folks (some of Popek's guys) had built a stackable file system
layer for BSD (which I think is in the contributed stuff in 4.4 - the
original BSD union FS and the UCLA replication FS for FICUS used it IIRC).
Some of the UMICH folks were messing with enhancements to NFS and it was an
active time.
Under the aegis of Unix International (UI), Gerry sponsored a pow-wow at
LCC (in the LA office by the airport) to talk about how stacking and VFS
layers. IIRC Kleinman from then NetApp was there. Besides' Dave, I've
forgotten now who else from Sun was there. There must have been someone
from USL and maybe Mical Gien of Chorus. But the whole discussion came
back to having FSS/VFS and N different DFS's, PFS's and layers like
replication or union, *etc* in everyone's kernel. Gerry saw there was
going to be a 'market' for support of different filing schemes, but at that
point, everything needed to be 'ported' because they were all different.
But also this idea of stacking was appealing. If we could come up with
something that clean, then maybe we could be more independent of the
different physical file systems and we could 'push' behavior into the stack
more invisible.
But to do we needed to get some sort of agreement and all use the same
thing.
USL (Sun) was insisting that a Sun VFS++ -- one interface to rule them all
- NFS was the standard already of course. This was not going over with the
Academic (UCLA/BSD/UMICH folks IIRC), particularly because of the lack of a
stacking property.
Dave proposed an idea of a general interposition layer on the inode that
would allow pushing and popping routines similar to the way STREAMS
worked. Which we landed on -- everybody had to change a little. I had
thought Dave implemented in an SVR4 kernel at some point and Chorus may
have had it. There was talk of trying to get the spec out of UI and LCC
was acting either as an arms merchant or the swiss, was trying to get
IBM/DEC/HP/OSF to come along. I went looking in my files and so far have
not been able to dig it up. But you might find something hunting around
the LOCKSS work as Dave was the mastermind behind it.
Eventually, is all blew up and I'm not sure what happened to either the
specs or the code.
Clem
On Tue, Jun 23, 2020 at 4:39 PM Rob Pike <robpike(a)gmail.com> wrote:
For my taste, the various Unix file system switches
that I've seen are too
firmly tied to the idea of blocks and disks and all that, making them less
flexible than they should have been. That's why the Plan 9 version is about
names and byte streams, to make it as general as possible.
That's one of the reasons the possibilities of the file system approach to
data has not reached its potential in Unix.
-rob
On Wed, Jun 24, 2020 at 1:14 AM Clem Cole <clemc(a)ccc.com> wrote:
> I agree with Larry with his observations. The only thing I would add is
> that Popek and Walker had a file system in the UCLA Locus distributed
> system for the 11/70 that was published in SIGOPS in late 1970s. In fact,
> Gerry came up with the idea after a sabbatical @ PARC. But it was not
> formalized like NFS as a separate (layered) FS, it was just part of the
> basic Locus FS.
>
> Peter Weinberger created the file system switch (FSS) for V8 and in the
> UNIX world, it was first. Dave Arnovitz used it for RFS in System V,
> while Perry Flynn and I used it for EFS, both of us after talking
> to Peter. Rusty and team did the VFS layer Sun @ (Larry will have to tell
> you who actually built it, I never knew). Rusty and I both published
> papers in the '85 Summer USENIX (NFS and EFS were contemporaries -- the
> difference is that Sun gave away NFS and Masscomp was not smart enough to
> do that).
>
> I talked to Steve Kleinman extensively about VFS at one point, and I'm
> pretty sure he and the rest of the Sun guys had talked to the PARC folks
> who after Gerry went back to UCLA started working on a DFS. The idea of
> the state-less (idempotent) file system RPC that NFS used based on stuff
> PARC did. But I'm not sure PARC had anything like the FSS or the VFS
> layer. Peter, Dave, and I used a stateful scheme because we chose to have
> full UNIX FS semantics, which NFS did not. In the end, early NFS was
> notorious for putting 'holes' in the files because of the automatic seek
in
> every operation and errors not coming until close(2) time.
>
> EFS used an RPC and a RUDP layer that Perry and Alan Atlas built, but it
> was not nearly as flexible as what Sun built [which had a crude interface
> generator], although until years later when Mike Leibensger built PIG (the
> Paceline Interface Generator) NFS RPC was always a PITA and not much better
> in practice than what we had at Masscomp.
>
> In fact, the point of the EFS paper was it's all about the recovery when
> there is a failure/error. If you read his paper, Rusty's point was who
> cares if there is an error (I've always felt vindicated that while I lost
> the war, over time everyone came to our way of thinking and now NFS
> V4 looks a whole lot like EFS did).
>
> Having a DFS as a feature was an incredible advance and proved we needed
> something (and it needed to be standard in all systems). NFS really lead
> the market with that advance, although it sort of took a few years to make
> it really good. The fact is that others had the same idea before or at
> least contemporary with it.
>
> That said, and to give the NFS team* a huge amount of credit *(and great
> applause), the VFS layer was better thought out than the FSS and in fact
> made it possible to add a lot of different file systems into UNIX later.
> FSS was much more ad hoc.
>
> At LCC, when we built VPROC for TNC a few years later, we used some of
> the same ideas from VFS and of course used VFS for the file system layer
> since TNC had to have full POSIX semantics. (It's a shame VPROC never
> caught on the way VFS did).
>
> Clem
>
>
> On Tue, Jun 23, 2020 at 10:02 AM Larry McVoy <lm(a)mcvoy.com> wrote:
>
>> I like to claim credit for creating the ChangeSet concept, the grouping
>> of a set of deltas across a number of files. But I wasn't the first.
>> Back when dejanews was a thing and you could search usenet posts in a
>> date range, if you searched before I started talking about ChangeSets,
>> you could find 6, count 'em, 6 hits. There was a really obscure system
>> called Aide De Camp, that had a similar concept.
>>
>> But if you searched after I started talking about them you would see
>> millions of hits.
>>
>> So I didn't invent the concept but I sure as heck made the world
>> understand the concept.
>>
>> I suspect Sun could be in a similar position. The VFS concept is
>> pretty sweet so there might have been someone before Sun. I'll
>> long odds that if there was, it didn't gain traction until Sun
>> did it.
>>
>> If there is nothing that predates it, then the inspiration was
>> almost certainly the device driver interface. One interface,
>> many devices. VFS is the same.
>>
>> On Tue, Jun 23, 2020 at 11:09:57AM +0200, Paul Ruizendaal wrote:
>> > When googling for File System Switch or Virtual File System most
>> sources mention Sun NFS and SysVr3 as the earliest implementations. Some
>> sources mention 8th Edition.
>> >
>> > I did a (short) search on FSS/VFS in earlier, non-Unix OS???s (Tenex,
>> Multics, CTSS, etc.), but none of those seem to have had a comparable
>> concept.
>> >
>> > Does anybody recall prior art (prior to 1984) in this area?
>> >
>> > Paul
>>
>> --
>> ---
>> Larry McVoy lm at mcvoy.com
>> http://www.mcvoy.com/lm
>>
>
9:36 p.m.
On Tue, Jun 23, 2020 at 11:12:40AM -0400, Clem Cole wrote:
Rusty and team did the VFS layer Sun @ (Larry will
have to tell
you who actually built it, I never knew).
Steve Kleiman is the only author on the VFS paper so I'm guessing that
was him.
Rusty was NFS.
Joe Moran (sun!mojo if I remember correctly) did the VM implementation,
it was co-designed by Joe, Shannon, and Gingell.
Side story: part of the reason I wanted to go to Sun was because when
I was a CS student at UW-Madison, Joe was a little older than me, and
I sat and watched him do a BSD port to a 68K (I think) in a little more
than 24 hours (sketchy on the time, I think that's right, almost positive
it was more than 20 hours, it was a chunk of time). As a guy who wanted
to be next to the metal, it was impressive.
In the end, early NFS was
notorious for putting 'holes' in the files because of the automatic seek in
every operation and errors not coming until close(2) time.
You have no idea how many of those holes that 16 bit SCCS checksum has
found (BitKeeper kept it).
EFS used an RPC and a RUDP layer that Perry and Alan
Atlas built, but it
was not nearly as flexible as what Sun built [which had a crude interface
generator], although until years later when Mike Leibensger built PIG (the
Paceline Interface Generator) NFS RPC was always a PITA and not much better
in practice than what we had at Masscomp.
Oh, I don't know if I agree. I took the RPC mechanism and vectorized it.
I probably have the code somewhere. It was pretty neat, 10Mbit was the
common networking and I did RPC Vector calls to the rstatd on literally
every machine in the Mountain View campus. It was thousands of machines
and it worked. The code split the RPC back into send and receive and
the code had to start gathering up answers before it was finished
sending the questions or the packets would be dropped.
It worked surprisingly well. Ron Minnich and Gene Kim made an NFS client
that worked as demo-ware. There is a paper about below, note that I'm
co-author only because I wrote the RPC vector code, I believe Gene did
the in kernel code, it was Ron and Gene's project.
http://mcvoy.com/lm/papers/bigfoot.pdf
In fact, the point of the EFS paper was it's all
about the recovery when
there is a failure/error. If you read his paper, Rusty's point was who
cares if there is an error (I've always felt vindicated that while I lost
the war, over time everyone came to our way of thinking and now NFS
V4 looks a whole lot like EFS did).
Yeah, they were sort of cavalier about errors in public. The weird
thing is Sun ran their entire network on Sun machines. Everything was
NFSv2 when I got there, the routers were Sun workstations, the servers
were Sun servers, the clients were Sun workstations. They ate their own
dogfood and NFS just worked, it was pretty unheard of to hear about NFS
issues other than performance tuning (stupid biods etc).
Having a DFS as a feature was an incredible advance
and proved we needed
something (and it needed to be standard in all systems). NFS really lead
the market with that advance, although it sort of took a few years to make
it really good. The fact is that others had the same idea before or at
least contemporary with it.
NFS was solid in v2 at Sun. I went to SGI and NFS was complete garbage.
SGI's networking principle engineer told me, when I complained about
NFS not being reliable, "That's what rcp is for". I was flabbergasted,
the networking group generated $18M/year in NFS sales.
It wasn't just SGI, every vender other Sun had a crappy NFS implementation.
I'm surprised NFS didn't have a worse rep because while NFS worked on Suns,
it really didn't work anywhere near as well elsewhere.
That said, and to give the NFS team* a huge amount of
credit *(and great
applause), the VFS layer was better thought out than the FSS and in fact
made it possible to add a lot of different file systems into UNIX later.
FSS was much more ad hoc.
The VFS layer was Steve, NFS was Rusty.
At LCC, when we built VPROC for TNC a few years later,
we used some of the
same ideas from VFS and of course used VFS for the file system layer since
TNC had to have full POSIX semantics. (It's a shame VPROC never caught on
the way VFS did).
VPROC was very cool and a requirement for any sort of distributed single
system image OS. I agree it is a shame that it didn't catch on.
--lm
25 Jun
25 Jun
8:52 a.m.
On 6/24/2020 12:36 PM, Larry McVoy wrote:
On Tue, Jun 23, 2020 at 11:12:40AM -0400, Clem Cole
wrote:
Larry's enumeration of NFS/VFS players was good.
I'd add Bob Lyon who led the NFS team (and who later founded Legato
along with Rusty, Joe Moran and Jon Kepecs -- all except Bob being UW
alumni.)
Bob came to Sun via PARC and brought with him a number of the ideas that
Clem had referenced earlier in this thread about RPC, etc. Although he's
on few papers his influence on defining and building NFS etc. was
essential.
Another important contributor was David Goldberg, for building the YP
services that, while functionally separate, enabled the administrative
and deployment environment that helped make NFS useful. David also wrote
the floating-point paper in the appendix to the Hennessy-Patterson
architecture textbook.
5 Jul
5 Jul
2:05 a.m.
On Wed, 24 Jun 2020, Larry McVoy wrote:
Aren't holes part of the file system semantics? Seek beyond EOF, write a
block, and you've created unallocated holes. Of course, they are filled
with zeroes as soon as you copy the file... I did fiddle with "cp" to
detect those holes and preserve them, but gave up for some reason
(performance hit in checking for all-zero blocks, I think, along with some
weird problem with the last block being all zeroes).
I used NFS when it first appeared and got bitten quite badly; I've never
really trusted it ever since, but I understand that it's vastly improved.
-- Dave
In the end,
early NFS was notorious for putting 'holes' in the files
because of the automatic seek in every operation and errors not coming
until close(2) time.
You have no idea how many of those holes that 16 bit SCCS checksum has
found (BitKeeper kept it).
2:16 a.m.
On Sun, Jul 05, 2020 at 10:05:57AM +1000, Dave Horsfall wrote:
On Wed, 24 Jun 2020, Larry McVoy wrote:
Aren't holes part of the file system semantics?
I'm not talking about legit holes, I'm talking about where your data used
be served up as a list of zeros.
The SCCS checksum is weak but kinda handy. You could see single bit errors
with it (at least you could in BitKeeper). It would say
checksum error: wanted %d, got %d
and you could look at the numbers and see that it was one bit off.
We found a bunch of those because computers used to be ECC or parity and
then 15 years ago or so, they just dumped the parity bit so single bit
errors go unreported (noice, computer industry).
In the
end, early NFS was notorious for putting 'holes' in the files
because of the automatic seek in every operation and errors not coming
until close(2) time.
You have no idea how many of those holes that 16 bit SCCS checksum has
found (BitKeeper kept it).
6 Jul
6 Jul
6:42 a.m.
On Sat, 4 Jul 2020, Larry McVoy wrote:
Ah; my mistake... Too much blood in my coffee stream.
Aren't
holes part of the file system semantics?
I'm not talking about legit holes, I'm talking about where your data
used be served up as a list of zeros. The SCCS checksum is weak but kinda handy. You could
see single bit
errors with it (at least you could in BitKeeper). [...]
I used to edit SCCS files if I stuffed up an update (and didn't want to
make another update); I discovered that if I zeroed the checksum then it
would be recalculated... I have no idea what happened should the
"genuine" checksum turn out to be zero.
-- Dave
6:51 p.m.
On Sun, Jul 05, 2020 at 10:05:57AM +1000, Dave Horsfall
wrote:
We found a bunch of those because computers used to be ECC or parity and
then 15 years ago or so, they just dumped the parity bit so single bit
errors go unreported (noice, computer industry).
To be fair, there are still some ECC systems. Unfortunately most
of the home-use Intel boxes aren't. My own home-use box isn't and
(now 6+ years ago) I had bad RAM in it that produced single-bit
errors in an inode block that led to panics that led to me finding
the single-bit errors, but I don't know if there are some damaged
files.
I keep thinking I'll replace it with a new box that does have ECC,
but haven't gotten around to it yet. I see some consumer-priced
AMD CPUs have at least theoretical ECC support but I haven't found
anything that says the ECC actually works, and have seen a few
articles that hint that it doesn't. iX sells NAS boxes that do
have ECC, though.
Chris
7 Jul
7 Jul
1:23 a.m.
New subject: ECC memory
Chris Torek <torek(a)torek.net> wrote:
I keep thinking I'll replace it with a new box
that does have ECC,
but haven't gotten around to it yet. I see some consumer-priced
AMD CPUs have at least theoretical ECC support but I haven't found
anything that says the ECC actually works, and have seen a few
articles that hint that it doesn't.
All the AMD Ryzen CPUs and chipsets have built-in ECC. It's easy since
the CPU pins talk directly to main memory. This is one among thousands
of reasons to avoid buying Intel CPUs.
The machine I'm typing on has ECC memory, and I bought it in 2019. You
have to pick a motherboard that wasn't designed by dolts. (I got the
Gigabyte Aorus "AX-370-GAMING5", which of course is no longer
manufactured.) And you have to spend an extra $5 or $10 on your DIMMs.
Get the motherboard maker's "Qualified Vendor List", e.g.:
http://download.gigabyte.us/FileList/Memory/mb_memory_ga-ax370-gaming5_pinn…
Be sure there *are* some approved ECC dimms on the list. Buy those.
You may or may not have to fiddle something in the BIOS settings.
Check dmesg when you first boot the machine (booting the installer is
fine). Make sure the Linux kernel sees the DIMMS, isn't subverted by
the BIOS, and understands the CPU/memory control registers. When it
works, you'll see something like:
[ 0.180161] EDAC MC: Ver: 3.0.0
[ 9.389338] EDAC amd64: Node 0: DRAM ECC enabled.
[ 9.389339] EDAC amd64: F17h detected (node 0).
[ 9.389375] EDAC MC: UMC0 chip selects:
[ 9.389376] EDAC amd64: MC: 0: 0MB 1: 0MB
[ 9.389376] EDAC amd64: MC: 2: 4096MB 3: 4096MB
[ 9.389378] EDAC MC: UMC1 chip selects:
[ 9.389379] EDAC amd64: MC: 0: 0MB 1: 0MB
[ 9.389379] EDAC amd64: MC: 2: 4096MB 3: 4096MB
[ 9.389380] EDAC amd64: using x8 syndromes.
[ 9.389380] EDAC amd64: MCT channel count: 2
[ 9.389422] EDAC MC0: Giving out device to module amd64_edac controller F17h: DEV
0000:00:18.3 (INTERRUPT)
[ 9.389428] EDAC PCI0: Giving out device to module amd64_edac controller EDAC PCI
controller: DEV 0000:00:18.0 (POLLED)
[ 9.389429] AMD64 EDAC driver v3.5.0
Success! Non-flaky main memory in a PC clone! Cost: about $50 plus
paying attention.
John
3:07 a.m.
On Jul 6, 2020, at 9:51 AM, Chris Torek <torek(a)torek.net> wrote:
For my fileserver I am using a Gigabyte X470 Aorus Gaming board
with "qualified" ECC RAM (dmidecode confirms this) but I am not
sure if FreeBSD has ECC support for such boards. At least I don't
know how to tell!
What I really want is a laptop with ECC to use as the fileserver
to avoid having to deal with UPS that lasts a few minutes, cabling
and suboptimal support from NUT. SSDs are fast enough for me.
Laptop batteries last long and are easy to replace and easy to
test when to shutdown the laptop. And less heat and space!
On Sun, Jul 05, 2020 at 10:05:57AM +1000, Dave
Horsfall wrote:
We found a bunch of those because computers used to be ECC or parity and
then 15 years ago or so, they just dumped the parity bit so single bit
errors go unreported (noice, computer industry).
To be fair, there are still some ECC systems. Unfortunately most
of the home-use Intel boxes aren't. My own home-use box isn't and
(now 6+ years ago) I had bad RAM in it that produced single-bit
errors in an inode block that led to panics that led to me finding
the single-bit errors, but I don't know if there are some damaged
files.
I keep thinking I'll replace it with a new box that does have ECC,
but haven't gotten around to it yet. I see some consumer-priced
AMD CPUs have at least theoretical ECC support but I haven't found
anything that says the ECC actually works, and have seen a few
articles that hint that it doesn't. iX sells NAS boxes that do
have ECC, though.
5 Jul
5 Jul
3:43 a.m.
On Sat, Jul 4, 2020 at 8:07 PM Dave Horsfall <dave(a)horsfall.org> wrote:
Aren't holes part of the file system semantics?
Exactly - and that was the problem with NFS. Consider two write
operations. Remember each op is a complete operation with a seek to where
it's going. If the first fails, but the error is not reported (NFS returns
errors on close), the second operations seek over the failed write -- UNIX
puts zeros in the file. File closes later and the size is fine of
course. Oh yeah, whoever bothered to check for errors on close (like the
traditional SCCS or RCS commands)?
Later to try to read your file back -- it will have a bunch of zeros.
As Larry says, running a simple checksum could catch a lot of these.
Anyway, I'm going to be good and lay off a diatribe on NFS. It sort of
worked 'good enough.' But I will say other systems (like AFS) were much
better, in practice, but it lost the war.
Clem
4:43 p.m.
On Sat, Jul 04, 2020 at 09:43:04PM -0400, Clem Cole wrote:
On Sat, Jul 4, 2020 at 8:07 PM Dave Horsfall
<dave(a)horsfall.org> wrote:
So I've encountered lots of holes in NFS files where there shouldn't be
any. So it is/was a thing. But that said, I can't remember a single
case of encountering that on Sun's campus. I don't know if my memory
is failing me, but I do know that when I left Sun and started working
with other NFS implementations, yeah, lots of problems. Somehow Sun
got it right where other people didn't.
The point I'm trying to make is that I don't think NFS was broken by
design, it worked when it was Sun servers and Sun clients. Sun's
entire campus, 10's of thousands of machines, used NFS. Sun would
have screeched to a halt if NFS didn't work reliably all the time.
So it was possible to get it to work.
My guess is that other people didn't understand the "rules" and did
things that created problems. Sun's clients did understand and did
not push NFS in ways that would break it.
My memory may not be the greatest but I can still remember being
astonished when I first ran into people saying NFS didn't work.
It worked great for Sun and Sun's customers.
Aren't holes part of the file system
semantics?
Exactly - and that was the problem with NFS. Consider two write
operations. Remember each op is a complete operation with a seek to where
it's going. If the first fails, but the error is not reported (NFS returns
errors on close), the second operations seek over the failed write -- UNIX
puts zeros in the file. File closes later and the size is fine of
course. Oh yeah, whoever bothered to check for errors on close (like the
traditional SCCS or RCS commands)?
Later to try to read your file back -- it will have a bunch of zeros. 
8:40 p.m.
On 7/5/2020 10:43 AM, Larry McVoy wrote:
So I've encountered lots of holes in NFS files
where there shouldn't be
any. So it is/was a thing. But that said, I can't remember a single
case of encountering that on Sun's campus. I don't know if my memory
is failing me, but I do know that when I left Sun and started working
with other NFS implementations, yeah, lots of problems. Somehow Sun
got it right where other people didn't.
I can say personally, since the early 90's on SunOS, I never ever saw
this problem in a variety of environments. One being Nynex Science and
Technology where I did a consulting stint. 800+ node Sun 3/4 SunOS
workstation/server environment, basically everyone's desktop was a Sun
workstation for email, documentation, whatever. Another being a defense
contractor I was at for 7 years, they were all Sun for engineering
workstations and servers.
There is one possibility I just thought of, and that's if the first
write fails and then a context switch happens, if enough free space is
made available before the next context switch back to the second write,
I can see that being a problem ;)
As if you weren't already tired of my rambling... When it comes to
non-Sun operating systems, all bets were off. They all (mostly) worked
with their own kind. That usually wasn't the case when it came to
cross-vendor support. Sun<->HP was not great, but that also may have
been driver problems in the one instance I tried it for an extended
period of time. Two instances at different customers of AIX<->Sun
actually worked rather well. The YP integration was key.
The entire problem with "holey" files and NFS is certainly related to
the usage type of the system in question. What was Sun doing? Email?
Software development? Using a common NFS share for the compiler? And
then copying their code up to a central location? Not a lot of
sync/write/sync/write activity, unless object file generation is a lot
of skipping around all over the place. ;)
art k.
10:08 p.m.
On Sun, Jul 5, 2020 at 10:43 AM Larry McVoy <lm(a)mcvoy.com> wrote:
My guess is that other people didn't understand
the "rules" and did
things that created problems. Sun's clients did understand and did
not push NFS in ways that would break it.
I >>believe<< that a difference was file I/O was based on mmap on SunOS and
not on other systems (don't know about Solaris). The error was handled by
the OS memory system. You tell me about how SGI handled I/O. Tru64 used
mmap and I think macOS does also from the Mach heritage. RTU/Ultrix was
traditional BSD. Stellix was SRV3. Both had a file system cache with
write-behind.
I never knew for sure, but I always suspected that was crux of the
difference in how/where the write failure were handled. But as you pointed
out, many production NFS sites not running Suns had huge problems with
holes in files that were not discovered until it was too late to fix them.
SCCS/RCS repositories were particularly suspect and because people tried to
use them for shared development areas, it could be a big issue.
10:42 p.m.
I always used the design principle "Write locally, read over NFS". This
obviated locking issues and fit in with the idea of fate-sharing: a write
would always succeed, even if reading would have to wait until R (the
machine doing the reading) was up. The only additional thing I needed was
the ability for W (the machine doing the writing) to notify R that
something had changed, which I did by having R run a process that listened
on a port that would be opened and then closed by W: no data flowed over
this connection. If this connection could not be made, the process on the
W side would loop in bounded exponential backoff.
On Sun, Jul 5, 2020 at 4:09 PM Clem Cole <clemc(a)ccc.com> wrote:
On Sun, Jul 5, 2020 at 10:43 AM Larry McVoy <lm(a)mcvoy.com> wrote:
My guess is that other people didn't
understand the "rules" and did
things that created problems. Sun's clients did understand and did
not push NFS in ways that would break it.
I >>believe<< that a difference was file I/O was based on mmap on SunOS
and not on other systems (don't know about Solaris). The error was
handled by the OS memory system. You tell me about how SGI handled I/O.
Tru64 used mmap and I think macOS does also from the Mach heritage.
RTU/Ultrix was traditional BSD. Stellix was SRV3. Both had a file
system cache with write-behind.
I never knew for sure, but I always suspected that was crux of the
difference in how/where the write failure were handled. But as you
pointed out, many production NFS sites not running Suns had huge problems
with holes in files that were not discovered until it was too late to fix
them. SCCS/RCS repositories were particularly suspect and because people
tried to use them for shared development areas, it could be a big issue.
11:04 p.m.
On Sun, Jul 5, 2020 at 4:42 PM John Cowan <cowan(a)ccil.org> wrote:
I always used the design principle "Write
locally, read over NFS".
This was the basic idea of AFS. Originally, the CMU folks did whole file
caching, but by AFS 4.0 time, they had a Locus token manager (think DLM)
that scaled really well so partial caching was allowed. It actually made a
small disk system possible. What tended to happen, on your first boot, of
course, you had to fill /bin and lot of heavily used directories. But
what happened is that your system quickly had only the files you really
needed on the local disk. - the ones you were writing, and the few you used
over and over.
FWIW: I know a couple of people that still run it. I ran it until a few
years ago when I switched NAS units just for cost reasons.
11:14 p.m.
On Sun, Jul 5, 2020 at 5:06 PM Clem Cole <clemc(a)ccc.com> wrote:
On Sun, Jul 5, 2020 at 4:42 PM John Cowan
<cowan(a)ccil.org> wrote:
There was a neat paper out of CERN a few years ago about how they're
turning down their AFS (now OpenAFS) cells.
https://iopscience.iop.org/article/10.1088/1742-6596/898/6/062040/pdf
It seems that the idea of a big, shared, distributed file namespace is
sadly disappearing. I feel like most of the web-based replacements are not
as seamlessly integrated with my preferred toolset as what they're
replacing, but I've also become more and more acutely aware that I am not
the target audience for those things.
Certainly, real-time collaboration via e.g. Google Drive is pretty amazing
and very dynamic, particularly when paired with e.g. real-time video chat,
but it also forces one into a particular model of interaction that I've
spent most of the last three decades consciously avoiding but now find no
escape from.
- Dan C.
I always used the design principle "Write
locally, read over NFS".
This was the basic idea of AFS. Originally, the CMU folks did whole file
caching, but by AFS 4.0 time, they had a Locus token manager (think DLM)
that scaled really well so partial caching was allowed. It actually made a
small disk system possible. What tended to happen, on your first boot, of
course, you had to fill /bin and lot of heavily used directories. But
what happened is that your system quickly had only the files you really
needed on the local disk. - the ones you were writing, and the few you
used over and over.
FWIW: I know a couple of people that still run it. I ran it until a few
years ago when I switched NAS units just for cost reasons.
24 Jun
24 Jun
6:51 p.m.
Paul Ruizendaal <pnr(a)planet.nl> once said:
When googling for File System Switch or Virtual File
System most sources
mention Sun NFS and SysVr3 as the earliest implementations. Some sources
mention 8th Edition.
I did a (short) search on FSS/VFS in earlier, non-Unix OS’s (Tenex,
Multics, CTSS, etc.), but none of those seem to have had a comparable
concept.
Does anybody recall prior art (prior to 1984) in this area?
The concept of the "remote inode" was introduced in:
Luderer, G. W., Che, H., Haggerty, J. P., Kirslis, P. A., & Marshall,
W. T. (1981). A distributed UNIX system based on a virtual circuit
switch. ACM SIGOPS Operating Systems Review, 15(5), 160-168.
Cheers,
Anthony
7:31 p.m.
The only other prior art I know of is
Toda, I. W. A. O. (1980). DCNA higher level protocols. IEEE
Transactions on Communications, 28(4), 575-584.
wherein they state the following:
A virtual file system, from the viewpoint of application programs on a
computer, models the file system functions of other computers. This is
done in the same way as with virtual terminals, a virtual file system
consists of a UPP having virtual files (VF), and a UCP which executes
virtual file system protocols.
I'd be interested if you find anything earlier.
Anthony
8:31 p.m.
On 24 Jun 2020, at 18:51, Anthony Martin <ality(a)pbrane.org> wrote:
Paul Ruizendaal <pnr(a)planet.nl> once said:
When googling for File System Switch or Virtual
File System most sources
mention Sun NFS and SysVr3 as the earliest implementations. Some sources
mention 8th Edition.
I did a (short) search on FSS/VFS in earlier, non-Unix OS’s (Tenex,
Multics, CTSS, etc.), but none of those seem to have had a comparable
concept.
Does anybody recall prior art (prior to 1984) in this area?
The concept of the "remote inode" was introduced in:
Luderer, G. W., Che, H., Haggerty, J. P., Kirslis, P. A., & Marshall,
W. T. (1981). A distributed UNIX system based on a virtual circuit
switch. ACM SIGOPS Operating Systems Review, 15(5), 160-168. The only other prior art I know of is
Toda, I. W. A. O. (1980). DCNA higher level protocols. IEEE
Transactions on Communications, 28(4), 575-584.
wherein they state the following:
A virtual file system, from the viewpoint of application programs on a
computer, models the file system functions of other computers. This is
done in the same way as with virtual terminals, a virtual file system
consists of a UPP having virtual files (VF), and a UCP which executes
virtual file system protocols.
Thanks for those pointers: I had not thought to include distributed systems in my initial
search. Clem also pointed out LOCUS in this area. It would seem that S/F-Unix (the Luderer
paper) and LOCUS happened at about the same time and both have the concept of a remote
inode.
The Luderer paper has only a few details about implementation and besides describing the
remote inode concept they mention:
“Of 27 system call types related to files, 18 result in message traffic if remote files
are involved. Of these, 10 contain a path name as an argument, and the remaining 8 refer
to already opened files. Path names or the data read or written can be up to 64K bytes
long. The structure of each message is a type code followed by type-dependent data.”
In a way, the approach resembles the system call forwarding that Heinz Lycklama did for
his 'satellite systems' work.
I asked Bill Marshall (one of the S/F co-authors) about it a few months ago and he recalls
that they ran into problems with corner case semantics in their implementation, and that
Peter Weinberger later approached the issue afresh 'thinking about the hard cases
first.' Bill does not have S/F-Unix source code, and I have not been able to contact
Gottfried Luderer yet.
I have not yet been able to look at the DCNA paper yet.
I'd be interested if you find anything earlier.
I came across this 1979 paper by P.M. Lu from Bell Labs Naperville
https://ieeexplore.ieee.org/document/762533
The abstract says:
"RIDE (Resource-Sharing in a Distributed Environment) has been designed and
implemented for the UNIX operating system to provide sharing of remote files, remote
process invocation and interprocess communication in a distributed computing environment.
The system is designed to support a uniform interface for both local and remote access in
a network. The user programs can be executed in a single or multiple machine environment
without any program modification.”
I do not know what RIDE was and if its approach qualifies as a FSS precursor. Maybe it is
like 'Unix United / Newcastle Connection' in its approach.
It does seem an interesting paper in any case, as its front page includes: 'This
system was designed to replace the network interface programs that previously operated on
the SPIDER system.'
25 Jun
25 Jun
2:56 a.m.
On 6/24/2020 11:31 AM, Paul Ruizendaal wrote:
I came across this 1979 paper by P.M. Lu from Bell
Labs Naperville
https://ieeexplore.ieee.org/document/762533
The abstract says:
"RIDE (Resource-Sharing in a Distributed Environment) ...
RIDE seems very
reminiscent of the RSEXEC (Resource Sharing EXEC) done
on TENEX earlier in the 1970s. The implementation of RSEXEC involved
encapsulation of a process fork by its parent rather than an
object-oriented abstraction inserted behind an existing interface like
VFS is. Both are functionally transparent to their users but are kind of
going in opposite directions in implementation.
As with other mechanisms that have been referenced in this thread, these
illustrate the claim that "all problems in computer science can be
solved by [adding] another layer of indirection". I don't know that
that's absolutely true but I've certainly applied it a lot, and like a
lot of things, excessive application can create more problems.
(I never knew who was the first to utter the claim as I've heard it said
by lots of people. However a quick check shows an attribution to David
Wheeler in the form of the "Fundamental theorem of software engineering".)
https://en.wikipedia.org/wiki/Fundamental_theorem_of_software_engineering
RSEXEC was a pretty remarkable facility in action. It (approximately)
was like having a command interpreter for the aggregated TENEX resources
of the ARPAnet. It was at least partly intended to serve users of TIPs
(the terminal interface processors used to access hosts) by letting them
treat all TENEXes as a single resource and not worry about which one
they got hosted onto "this time" with respect to their files,
applications, etc., and importantly also not worry about the where their
collaborators were hosted at the moment.
A description of RSEXEC is at:
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.830.1403&rep=r…
The encapsulation mechanism was JSYS Traps, JSYS being the system call
instruction on PDP-10s modified to run TENEX. A description of that is at:
https://archive.org/details/bitsavers_bbntenexTh_1137399
In it's way, RSEXEC illustrated one characteristic of many excellent
innovations in that when used as intended it was "deceptively
unimpressive". As in: harder to demo.
NFS and other things share this trait, and indeed early trade show
demonstrations of NFS were frustrating (in those days, the engineers did
booth duty). At first, demos reduced to typing "ls". Which,
unsurprisingly, produced a list of files. It's not until you know how it
was done that it's impressive.
There can be a real utilitarian pleasure in things that, when shown to
others, produce a "yeah, wasn't it supposed to do that?" reaction. Part
of that aesthetic appreciation for me originated with RSEXEC. Which
brings me to VFS and such.
Earlier in this thread Clem Cole observed:
On 6/23/2020 8:12 AM, Clem Cole wrote:
That said, and to give the NFS team_a huge amount of
credit _(and great
applause), the VFS layer was better thought out than the FSS and in fact
made it possible to add a lot of different file systems into UNIX
later. FSS was much more ad hoc.
I think VFS and several related things are another instance of that
aesthetic appreciation. [I'm not claiming any credit, the initial NFS
work was just finishing up as I arrived at Sun. That work like that
happened there was part of why I wanted to be there and so I share
Clem's appreciation for the team.]
NFS could have been implemented by sprinkling "if (network)" statements
around a lot of code. A user or application program would not know any
differently had it been done that way. (Though they would over time as
the code became less maintainable with that implementation approach.)
That VFS has the qualities leading to Clem's applause is I think due to
a confluence of several things that likely hadn't occurred previously
and thus entitle it to claim "first" in response to the thread-starting
question.
One of the motivators was the desire to have a network filing
abstraction, and to replace "nd" for diskless workstations among other
things. (That took a while, "nd" really hung on for various reasons.)
That's occurring in an environment of heterogeneity and "open systems"
thinking (which is a different thing from "open source"), and the desire
to be more "of the network" in the product offerings.
There was kind of a group-innate understanding that to be successful in
introducing these capabilities that they had to occur largely "behind"
the extant interfaces used by the nascent UNIX application market. To
some extent Sun's market success was based on being deliberately "more
vanilla than thou" with respect to APIs. The variations already extant
were a market-fragmenting influence and thus didn't need an additional
player doing more of it.
To be "of the network" made NFS a "file system for the Network (that
happens to look like UNIX)". The similarity and the known semantic
differences lead some to look at it as "it failed to do UNIX" but it's
not. The intent was "do the network, the network is NOT 'a' computer
(irony alert), the fallacies of networking apply and we resolve the
irreconcilable in favor of the network".
(An intangible factor relates to the group that Sun formed, talented
engineers, strikingly different people who shared good taste in design
and the desire to do the work to find the "simple" in a problem. A
different group of people with the same problem set wouldn't have
automatically gotten the same or even similar answer, sometimes it's
just in the air. Of course, a different group might have made an even
better answer, "innovation happens elsewhere" is more true than the
earlier-referenced layer of indirection principle.)
Sometimes (mostly even), good taste is expressed in framing the problem
appropriately. The "N" in NFS is deliberately not a "D". That's
obvious
from the resulting definition but earlier in the effort it was a choice.
The intended distinction was that we weren't doing operating system work
in order to transcend the network by submerging it in the abstractions
of an OS. Rather, we wanted to build a network environment populated by
systems which were of a piece with it. While not codified until much
later, Deutsch's "Fallacies of Networking" were the controlling notions
in a lot of thinking and influenced things that at first wouldn't seem
to have anything to do with networking (e.g., msync()).
However, it would be wrong to say we realized enough at the time to
*know* that our goal was to build a system defined by a metric that it
avoid all of the networking fallacies. We knew some of it, were
incrementally hunting it, instincts were oriented that way. Had we been
a research organization that might have given rise to an initiative that
was more clearly defined. (But, I think we did a reasonable job of
pursuing the ideas while making products that people who weren't
thinking of such things bought. As it happens, a lot of FORTRAN users as
has been noted here before.
Regarding the earlier irony alert: to the extent that the evocative
marketing slogan of "The Network is the Computer" had any technical
underpinnings, it was an expression of this aim to design in favor of
the network. It isn't really irony so much as consequence.
The real irony is that when the slogan was first trotted out almost all
of the company but particularly engineering barfed at it. I recall the
marketing person(s) (individual contributors, not the executives later
claiming credit) trying to push it receiving huge rations of abuse. But
it stuck. And we came around once we looked beyond its technical
deficiency as a literal statement and saw it as evocative of what we had
been doing. They knew us better than we did. And, independent of any
technical matter, it worked really well. 'Find the simple" is a useful
admonition for marketing as well as engineering.
6:15 a.m.
New subject: Oh, things were very different.
5:45 p.m.
New subject: Oh, things were very different.
On 2020, Jun 25, at 12:15 AM, Rich Morin
<rdm(a)cfcl.com> wrote:
https://imgs.xkcd.com/comics/old_days_2.png
-r
The trick to these is to include one nugget of truth. If anyone asks, I’m saying the
sweater part is absolutely true.
After all, BITNET actually did work by connecting a virtual card punch to a remote virtual
card reader.
-L
6:24 p.m.
New subject: Oh, things were very different.
On Thu, Jun 25, 2020 at 9:52 AM Lawrence Stewart <stewart(a)serissa.com>
wrote:
I have thousands of virtual sweaters in a drawer upstairs. They were what
inspired virtual machines and the cloud after all...
Warner
On 2020, Jun 25, at 12:15 AM, Rich Morin
<rdm(a)cfcl.com> wrote:
https://imgs.xkcd.com/comics/old_days_2.png
-r
The trick to these is to include one nugget of truth. If anyone asks, I’m
saying the sweater part is absolutely true.
After all, BITNET actually did work by connecting a virtual card punch to
a remote virtual card reader.
7:57 p.m.
New subject: Oh, things were very different.
On 6/25/2020 11:45 AM, Lawrence Stewart wrote:
The trick to these is to include one nugget of truth.
If anyone asks, I’m saying the sweater part is absolutely true.
After all, BITNET actually did work by connecting a virtual card punch to a remote
virtual card reader.
-L
I always knew Mr. Rogers was a deep state operative. All those
sweaters... probably generated by years of anti-encryption efforts...
10:22 p.m.
New subject: Oh, things were very different.
On Wed, 24 Jun 2020, Rich Morin wrote:
Got the original cartoon number for that? I like to see the comment when
I hover over it (and the explanation site can be funny, too).
-- Dave
10:42 p.m.
New subject: Oh, things were very different.
On 26 Jun 2020 06:22 +1000, from dave(a)horsfall.org (Dave Horsfall):
Got the original cartoon number for that?
It's brand new.
https://xkcd.com/2324/
The first one is https://xkcd.com/1755/
--
Michael Kjörling • https://michael.kjorling.se • michael(a)kjorling.se
“Remember when, on the Internet, nobody cared that you were a dog?”
10:49 p.m.
New subject: Oh, things were very different.
2324
On Thu, Jun 25, 2020 at 4:23 PM Dave Horsfall <dave(a)horsfall.org> wrote:
On Wed, 24 Jun 2020, Rich Morin wrote:
Got the original cartoon number for that? I like to see the comment when
I hover over it (and the explanation site can be funny, too).
-- Dave
26 Jun
26 Jun
2:48 a.m.
New subject: Oh, things were very different.
On Thu, 25 Jun 2020, Clem Cole wrote:
Thanks! The hover-over makes a bit more sense now, and explainxkcd.com is
fascinating.
-- Dave
Got the
original cartoon number for that? I like to see the comment
when I hover over it (and the explanation site can be funny, too).
2324
24 Jun
24 Jun
9:05 p.m.
I came across this 1979 paper by P.M. Lu from Bell
Labs Naperville
Full paper is available here:
https://www.computer.org/csdl/pds/api/csdl/proceedings/download-article/12O…
I do not know what RIDE was and if its approach
qualifies as a FSS precursor. Maybe it is like 'Unix United / Newcastle
Connection' in its approach.
Indeed it appears to have been like the (later) Newcastle Connection in its approach.
10:27 p.m.
On Tue, Jun 23, 2020 at 11:09:57AM +0200, Paul Ruizendaal wrote:
I did a (short) search on FSS/VFS in earlier,
non-Unix OS’s (Tenex, Multics, CTSS, etc.), but none of those seem to have had a
comparable concept.
Does anybody recall prior art (prior to 1984) in this area?
Surely any OS, even non unix, which offered a remote FS in addition to a local FS
implicitly implements something like a FSS/VFS?
So we could look to CP/M (and CP/Net, CP/NOS), where this[1] seems to suggest a date
between 1980 and 1982.
An appendix to that paper has the CPM file system operations (such as they were) remotely
available and
listed with their codes for the messages.
That said, another OS from DR (FlexOS) has a recognisable VFS, is written in C, but dates
to early 1985.
I'm not sure when the design dates from[3]. If one looks at this[2] article in Byte,
each of the Resource Managers operates through a VFS layer, in terms of:
#define I_GET 3 /* NET DISK CON MSC PIPE KERN */
#define I_SET 4 /* NET DISK CON MSC PIPE KERN */
#define I_LOOKUP 5 /* NET DISK CON MSC PIPE KERN */
#define I_OPEN 6 /* NET DISK CON MSC PIPE */
#define I_CLOSE 7 /* NET DISK CON MSC PIPE */
#define I_READ 8 /* NET DISK CON MSC PIPE */
#define I_WRITE 9 /* NET DISK CON MSC PIPE */
#define I_SEEK 10 /* NET DISK CON MSC PIPE */
#define I_CREATE 11 /* NET DISK CON PIPE */
#define I_DELETE 12 /* NET DISK CON PIPE */
Which can be seen in the public code for CDOS-68k.
Not all RMs implement all calls, but the NET, DISK and PIPE sort of did.
DF
[1] https://www.autometer.de/unix4fun/z80pack/cpnet/cpnet.htm
[2] http://tech-insider.org/personal-computers/research/1985/05.html
https://archive.org/stream/byte-magazine-1985-05/1985_05_BYTE_10-05_Multipr…
[3] The 80286 was released in '82, but I don't know when Intel started giving
early info to customers.
11:33 p.m.
At Tue, 23 Jun 2020 11:09:57 +0200, Paul Ruizendaal <pnr(a)planet.nl> wrote:
Subject: [TUHS] VFS prior to 1984
When googling for File System Switch or Virtual File System most
sources mention Sun NFS and SysVr3 as the earliest
implementations. Some sources mention 8th Edition.
I did a (short) search on FSS/VFS in earlier, non-Unix OS’s (Tenex,
Multics, CASS, etc.), but none of those seem to have had a comparable
concept.
Does anybody recall prior art (prior to 1984) in this area?
Well, with regard to Multics, it had what were called "Device Interface
Modules" (DIMs) for building device drivers to either hardware or
pseudo-devices (an example of a pseudo-DIM is the File System Interface
Module (FSIM) which associates a segment (file) in the filesystem (by
path name) with a stream, i.e. it implements I/O redirection).
DIMs had a well defined API, and as Organick said in his 1972 book:
"The DIM converts a device independent request into a device dependent
one." ... "The file-system interface DIM is used to make a segment look
like an I/O device."
A DIM also had a name, an identifier, and that's how you attached a
"stream" to it:
call attach (stream_name, DIM_name, device_name);
All of those parameters are strings.
As an aside I haven't found any evidence that DIM identifiers were ever
listed in the filesystem, e.g. as Unix would have in "/dev", but that
would be an obvious thing to do these days. Then they were just listed
in the programmer's manual, and presumably user-written ones were added
to the online documentation. Today such an idea would be "obvious".
As far as I can remember Multics didn't really have the concept of a
"mount point". All storage was single-level, i.e. segments (equivalent
in some respects to inodes, but they are also actually the value of the
segment register in the virtual memory hardware), and so files were
either physically in memory or paged out on physical disk devices or
similar, or even out on tape. Where they actually resided was entirely
and permanently hidden from the user. What was called the "filesystem"
was a form of database representing a hierarchical namespace which
pointed at all the known segments (files) regardless of where they were
actually stored.
In terms of virtual filesystems for Multics, I think it would be (have
been) relatively easy enough (i.e. with the addition of some APIs to the
filesystem itself) to implement a way of "mounting" a pseudo-DIM into
the filesystem namespace and thus being able to implement anything from
the likes of /kern and /proc to remote filesystems, network directories,
etc. I.e. create an inverse twin of the filesystem DIM that takes
filesystem requests and turns them into back-end pseudo-DIM requests.
Effectively this would be implementing a VFS for Multics. Indeed today,
and perhaps any time after Apollo Domain was designed, it would be
plainly obvious, and probably desirable, to do such a thing, at least
for some kinds of concepts, such as /proc.
Note though that I suspect the original Multics designers would have
preferred it if remote filesystems, in particular, were also effectively
invisible to the user, though I do not know how they might have
approached the issue of remote segments not always being available,
including indeed parts of the filesystem itself, i.e. directories and
metadata. Perhaps in a similar way to how files migrated to backup
storage were handled -- if you referenced a file that had been migrated
offline then your process trapped to a retrieval process that involved
prompting a human tape operator to load the appropriate backup tape(s).
In fact Organick says the following about this idea of an inverse
relationship between the filesystem and I/O subsystem in his 1972 book
(p351,352):
"We have just seen how the I/O system may behave as a customer of
the file system, which supplies needed services. One might wonder
if the reverse of those roles is ever true. That is, does the file
system through its Page Control Module, when seeking to transfer a
page of information to/from core [[memory]] and disk or tape
storage, ever find itself to be a customer of the I/O system?
Emphasis on objectives of modularity might cause one to guess the
affirmative. In actual fact however, considerations of efficiency
have dictated that paging I/O in Multics be treated with
special-purpose I/O software that greatly streamlines the
processor's task in initiating and controlling such I/O."
So, at least early on, he appears to have denied the idea of the
filesystem routing requests through a DIM or pseudo-DIM, though he did
so on the basis that this would be done at the pager level and that
doing so could be too inefficient to consider. However he doesn't seem
to have considered, or at least does not mention in the 1972 book, the
idea of a pseudo-DIM extending the filesystem namespace by implementing
a mount point, which as I say would today be considered a quite obvious,
desirable, and feasible task.
In any case the idea of pseudo device drivers interacting with the
filesystem, albeit the other way around, was firmly in place by 1965
(the file-system interface pseudo-DIM was described in the 1965 Multics
I/O system paper).
--
Greg A. Woods <gwoods(a)acm.org>
Kelowna, BC +1 250 762-7675 RoboHack <woods(a)robohack.ca>
Planix, Inc. <woods(a)planix.com> Avoncote Farms <woods(a)avoncote.ca>
25 Jun
25 Jun
2:45 a.m.
On Wed, Jun 24, 2020 at 2:34 PM Greg A. Woods <woods(a)robohack.ca> wrote:
As far as I can remember Multics didn't really have the concept of a
"mount point". All storage was single-level, i.e. segments (equivalent
in some respects to inodes, but they are also actually the value of the
segment register in the virtual memory hardware), and so files were
either physically in memory or paged out on physical disk devices or
similar, or even out on tape. Where they actually resided was entirely
and permanently hidden from the user. What was called the "filesystem"
was a form of database representing a hierarchical namespace which
pointed at all the known segments (files) regardless of where they were
actually stored.
Coming to it from a Unix perspective, it's like all storage (core, disk,
tape) is mmap()ed.
The segment-name database then is just an index relating symbolic names to
particular memory locations.
It all feels very upside-down to me, but that's probably because I grew up
in Unix and never actually used a Multics system until I emulated one with
dps8m.
Adam
9:40 p.m.
At Wed, 24 Jun 2020 17:45:38 -0700, Adam Thornton <athornton(a)gmail.com> wrote:
Subject: Re: [TUHS] VFS prior to 1984
On Wed, Jun 24, 2020 at 2:34 PM Greg A. Woods <woods(a)robohack.ca> wrote:
Yes, exactly. A process is just a collection of segments of (virtual)
memory, some containing code, some containing data, and more can be
attached, some can be detached and/or swapped, as execution proceeds.
E.g. for shared code, a module can be swapped out for another equivalent
one, e.g. with a bug fix, without stopping or restarting the process
(and perhaps, if it is a system library, without the user or application
even knowing that has happened).
As far as I can remember Multics didn't really have the concept of a
"mount point". All storage was single-level, i.e. segments (equivalent
in some respects to inodes, but they are also actually the value of the
segment register in the virtual memory hardware), and so files were
either physically in memory or paged out on physical disk devices or
similar, or even out on tape. Where they actually resided was entirely
and permanently hidden from the user. What was called the "filesystem"
was a form of database representing a hierarchical namespace which
pointed at all the known segments (files) regardless of where they were
actually stored.
Coming to it from a Unix perspective, it's like all storage (core, disk,
tape) is mmap()ed.
The segment-name database then is just an index relating symbolic names to
particular memory locations. It all feels very upside-down to me, but that's
probably because I grew up
in Unix and never actually used a Multics system until I emulated one with
dps8m.
I kind of thought of it that way in my early years too, even though
Multics was one of the first systems I learned, at the same time as I
learned Unix.
In Multics I/O takes a far lesser role than it does in Unix (or most any
other OS then and since).
Modeling everything as if it were I/O has some advantages in terms of
implementation of the model (I guess it makes for far simpler operating
systems), but personally I now think it's completely upside down and
inside out from the user's perspective.
Having to read all data before you can actually access it, and/or even
just having to manage your allotted memory space and pointer offsets, is
quite an annoying addition to any algorithm -- i.e. adding I/O in front
of everything is like adding a significant problem on top of every other
problem, every time, all the time. (Well using full-on only mmap() and
a good language and compiler could probably hide much of the nastiness,
but.... we don't really seem to do that even to this day, even in
languages that could make it easy.)
So, personally, I now see modeling everything as I/O as if it is forcing
you to access all your data through the eye of a needle, every time, all
the time. No matter how big the hole you still have to coil all your
data up and push/pull it through if it is, or could be, bigger than the
hole.
I'm also quite sad that the extremely wide and near pervasive and
long-term availability of a CPU family with half-decent support for
segmented virtual memory has effectively been wasted for a generation or
two or more of possibilities in OS design.
--
Greg A. Woods <gwoods(a)acm.org>
Kelowna, BC +1 250 762-7675 RoboHack <woods(a)robohack.ca>
Planix, Inc. <woods(a)planix.com> Avoncote Farms <woods(a)avoncote.ca>
1558
days inactive
1572
days old
40 comments
21 participants
participants (21)
-
Adam Thornton -
Anthony Martin -
arnold＠skeeve.com -
Arthur Krewat -
Bakul Shah -
Chris Torek -
Clem Cole -
Dan Cross -
Dave Horsfall -
Derek Fawcus -
Greg A. Woods -
John Cowan -
John Gilmore -
Larry McVoy -
Lawrence Stewart -
Michael Kjörling -
Paul Ruizendaal -
Rich Morin -
Rob Gingell -
Rob Pike -
Warner Losh