[TUHS] [tuhs] The Unix shell: a 50-year view

[Jon Steinhart]

Dan Cross writes:
>
> Interestingly, much of this is kind of the point I was trying to make
> earlier: many of the problems are different now. Trying to force them into
> a metaphor that made sense for the problems that were primary 40 years ago
> can lead to awkward and inefficient solutions. It doesn't always have to,
> but sometimes it does.
>
> "Everything is a file" and "files are streams of bytes" was great for
> approaching so many of the sorts of problems that were interesting in the 1970s
> and 80s. Now days, there are whole new types of problems that don't fit into
> that model easily; is that the fault of the problem, or a suggestion that we
> need to expand our model and reexamine the basic structure of our systems?
> Even, to some extent, the notion of a global hierarchical file namespace has
> broken down. Now, I want to be able to partition and organize datasets across
> multiple dimensions and across multiple machines and group things on an ad hoc
> basis; it's all very dynamic, and by comparison the model of a 7th Edition
> filesystem is static and downright limited.

Been trying hard to resist but here goes...

On the snarky side, yeah, problems are different now, and the shell paper
is a great example.  Such a bad paper winning a best paper award from
the ACM illustrates how the profession just ain't what it used to be.
I would claim that a large chunk of "software engineering" today involves
trying to build environments that minimize the amount of harm that can
be done by under-skilled practitioners.  I have difficulty envisioning
success as there's a heavy focus on memory management which to me is a
great competency test.  Just because one is coddled and can't corrupt
memory doesn't mean that the the remainder of their code is correct.
These things go hand-in-hand.

I'm going to stick my neck out and claim that to a large degree we're the
victims of our own success.  When I was writing my book, Clem pointed
out to me that a big assumption that Stallman made was that people
who would contribute to open source software would be of his caliber.
I don't think that he ever thought that we'd get to the point where
every programming class project would be published online and reused by
others without much inspection.  To me it's analogous to what happened
with MIDI when one could make "music" without having suffered for their
art like their predecessors.  Just decreased the signal-to-noise ratio
for me and made it much more cumbersome to find music that I liked.

Many of the problems that people seem to be trying to solve today are
of our own making.  I don't see the benefit of Linux distributions
being incompatible in annoying ways.  If each distro didn't put things
in different places for no discernible reason, we wouldn't need to
be putting as much non-productive energy into tools for deployment.
Likewise if there was any rhyme or reason for what's in what library
and more attention to API design and stability.  I kinda have to go
back to my 1989 Usenix panel session; given the choice between having a
good base and covering up a bad base with a pile of other code I choose
the first.  Sure, if we make stuff with tons of preventable dependencies
then we have to work with that, but I'd rather not have the problem in
the first place.

Another one - I recall when a lot of work went into making it possible
to link C and FORTRAN programs.  Instead of continuing along that path,
now we have multiple systems (PHP, Python, Perl, Java, ...) that have
each written their own libraries for everything.  To me, I would
have preferred to have a single set of libraries and the ability
to link.  Aside from the enormous waste of person-hours, there is a
huge non-intersecting set of bugs.  In theory we'd have fewer bugs
if more people were using the same libraries.  And better programmer
portability too.

I'm often surprised when people claim that the UNIX philosophy no
longer works.  I've been told "pipes only work for ASCII data" and have
gotten blank looks when I respond with "ImageMagick seems to work fine".
And "you can't do modern web crud" but then along came things like "jq".

To me, the UNIX philosophy beats the big monolithic program philosophy
every time.  Sure, these two don't interact when big monolithic programs
are crafted such that they don't play well with others but that's not
the fault of the UNIX philosophy, it's the fault of the monolithic
program design.  Watch Matt Blaze's 2016 congressional testimony if
you haven't already done so.  He makes a great case that we know that
we can't prove program correctness despite trying to do so since the
beginning of time, so the best security comes from simplicity which
minimizes the number of attack surfaces.

So rather than saying that the UNIX philosophy no longer works, how
about giving some serious thought to how to do what you want to do in
the context of the philosophy?  In particular, how to you extend the
existing tool set to accomplish your purpose instead of just throwing up
your hands and creating more big monolithic non-interoperable packages?
Certainly the first is harder, but lasts longer and tastes great too.
A question that I asked in my book was why ~0% of the 1200 pages of
"Inside Macintosh" from 1985 are used today compared to ~100% of the
323 pages of UNIX V6 from 1975.  My answer is "abstractions".  I don't
see any of the "modern packages" having that level of staying power.

There is a lot more to the UNIX philosophy than "Everything is a file"
and "files are streams of bytes"; it's a mindset.  I have no problem with
model expansion and structure reexamination.  But piling poorly thought
out crap on top of other poorly thought out crap isn't a solution to me.
What new problems are you trying to solve and what are the appropriate
abstractions (and how much of this was done in Plan 9 and ignored)?

Haven't thought about this in forever as it's several lifetimes ago.
Once upon a time I was responsible for implementing GKS (the useless
ISO Graphical Kernel System) on a workstation project and was
participating in the ANSI standardization effort.  At the time, there
was another standard (PHIGS - Programmer's Hierarchical Interface
to GraphicS) in the works; I remember asking why.  The answer was
that there was no way to add segment hierarchy and editing to GKS in a
compatible way.  In reality, the folks just wanted to do their own thing;
they were furious when I published a paper on how to easily add that
functionality to GKS.  It's easy to say that something can't be done;
harder to put in some serious thought to figure out how it can be done.

Bottom line is, just read the news.  Is today's software methodology
working?  Does the fact that Facebook is funding a PR campaign to
convince people that privacy breaches are normal tell you anything?
There's a lot of similarity between modern software and the Surfside
condo; shiny stuff on top of a crumbling foundation is bound to fail.
In lectures these days, I'm asking the question "We haven't managed
to off more than a thousand or so people at a time with a software bug
yet so what's going to be software's Union Carbide Bhopal moment?"

Jon Steinhart

P.S.  On another topic, GUIs were the killer app for threads.  Sure, I would
      much prefer faster process context switching.  Still waiting for that.