30 Dec
2022
30 Dec
'22
6:25 p.m.
London and Reiser report about porting the shell that “it required by far the largest
conversion effort of any supposedly portable program, for the simple reason that it is not
portable.” By the time of SysIII this is greatly improved, but also in porting the SysIII
user land it was the most complex of the set so far.
There were three aspects that I found noteworthy:
1. London/Reiser apparently felt strongly about a property of casts. The code argues that
casting an l-value should not convert it into a r-value:
<quote from "mode.h">
/* the following nonsense is required
* because casts turn an Lvalue
* into an Rvalue so two cheats
* are necessary, one for each context.
*/
union { int _cheat;};
#define Lcheat(a) ((a)._cheat)
#define Rcheat(a) ((int)(a))
<endquote>
However, Lcheat is only used in two places (in service.c), to set and to clear a flag in a
pointer. Interestingly, the 32V code already replaces one of these instances with a
regular r-value cast. So far, I’d never thought about this aspect of casts. I stumbled
across it, because the Plan 9 compiler did not accept the Lcheat expansion as valid C.
2. On the history of dup2
The shell code includes the following:
<quote from “io.c”>
rename(f1,f2)
REG INT f1, f2;
{
#ifdef RES /* research has different sys calls from TS */
IF f1!=f2
THEN dup(f1|DUPFLG, f2);
close(f1);
IF f2==0 THEN ioset|=1 FI
FI
#else
INT fs;
IF f1!=f2
THEN fs = fcntl(f2,1,0);
close(f2);
fcntl(f1,0,f2);
close(f1);
IF fs==1 THEN fcntl(f2,2,1) FI
IF f2==0 THEN ioset|=1 FI
FI
#endif
}
<endquote>
I’ve check the 8th edition source, and indeed it supports using DUPFLG to signal to dup()
that it really is dup2(). I had earlier wondered why dup2() did not appear in research
until 10th edition, but now that is clear. It would seem that the dup of 8th edition is a
direct ancestor to dup() in Plan 9. I wonder why this way of doing things never caught on
in the other Unices.
3. Halfway to demand paging
I stumbled across this one because I had a bug in my signal handling. From early days
onwards, Unix supported dynamically growing the stack allocation, which arguably is a
first step towards building the mechanisms for demand paging. It appears that the Bourne
shell made another step, catching page faults and expanding the data/bss allocation
dynamically:
<quote from “fault.c”>
VOID fault(sig)
REG INT sig;
{
signal(sig, fault);
IF sig==MEMF
THEN IF setbrk(brkincr) == -1
THEN error(nospace);
FI
ELIF ...
<endquote>
This was already present in 7th edition, so it is by no means new in 32V or SysIII -- it
had just escaped my attention as a conceptual step in the development of Unix memory
handling.
30 Dec
30 Dec
7:51 p.m.
On 12/30/22 1:25 PM, Paul Ruizendaal wrote:
London and Reiser report about porting the shell that “it required by far the largest
conversion effort of any supposedly portable program, for the simple reason that it is not
portable.” By the time of SysIII this is greatly improved, but also in porting the SysIII
user land it was the most complex of the set so far.
Have you read
http://www.collyer.net/who/geoff/sh.tour.pdf
and looked at http://www.collyer.net/who/geoff/v7sh.tar ?
In the limited literature on Bourne Shell porting, this is authoritative.
Arnold Robbins built on that work and ported the v8-v10 shells to modern
Linux versions. (I am sorry, I do not have a link right now.)
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU chet(a)case.edu http://tiswww.cwru.edu/~chet/
8:02 p.m.
On Fri, Dec 30, 2022 at 02:51:26PM -0500, Chet Ramey wrote:
On 12/30/22 1:25 PM, Paul Ruizendaal wrote:
Is there are reason to hang on to the Bourne shell? Maybe shell scripts?
Does it perform better than ksh or bash?
Don't get me wrong, I much prefer the sh syntax over csh syntax, but
I'd never go back to the Bourne shell as my login shell. Way too much
useful stuff in ksh/bash.
London and Reiser report about porting the shell that ???it required by far the largest
conversion effort of any supposedly portable program, for the simple reason that it is not
portable.??? By the time of SysIII this is greatly improved, but also in porting the
SysIII user land it was the most complex of the set so far.
Have you read
http://www.collyer.net/who/geoff/sh.tour.pdf
and looked at http://www.collyer.net/who/geoff/v7sh.tar ?
In the limited literature on Bourne Shell porting, this is authoritative. 
8:31 p.m.
On Dec 30, 2022, at 1:02 PM, Larry McVoy
<lm(a)mcvoy.com> wrote:
Is there are reason to hang on to the Bourne shell? Maybe shell scripts?
Does it perform better than ksh or bash?
POSIX sh seems like a good lowest-common-denominator shell to write for (sorry, SunOS and
xpg4).
Don't get me wrong, I much prefer the sh syntax over csh syntax, but
I'd never go back to the Bourne shell as my login shell. Way too much
useful stuff in ksh/bash.
I mean I would assume the reason is that bash takes a whole lot more memory and CPU to do
its very convenient magic. That's an issue if you're trying to run on a PDP-11
and only have 16 bits of address space, but in 64-bit-world, where you almost certainly
have at least half a gigabyte of core storage available, and your processor clock is
almost certainly in at least the high hundreds of MHz...yeah.
Although (because my daily driver is a Mac) I've finally taken the zsh plunge. I
feel a little dirty, but oh-my-zsh certainly has its attractions.
Adam
8:49 p.m.
On 12/30/22 3:31 PM, Adam Thornton wrote:
Although (because my daily driver is a Mac) I've
finally taken the zsh plunge.
You don't have to, you know. ;-)
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU chet(a)case.edu http://tiswww.cwru.edu/~chet/
8:42 p.m.
Larry McVoy on 30.12.2022 21:02:
I'd like the idea of "preserving a heirloom in its natural environment"
(and even more effort went in https://heirloom.sourceforge.net/sh.html)
let alone this does not prevent from adding modern shells...
I guess in interactive use most users would only miss one thing: the
history & line editing capability?
Side notes to that:
* By intention, the almquist shell (a port due to the Berkeley/ATT
mess) initially had no history. From the package file DIFFERENCES [1],
"History. It seems to me that the csh history mechanism is mostly
a response to the deficiencies of UNIX terminal I/O. Those of you
running 4.2 BSD should try out atty (which I am posting to the net
at the same time as ash) and see if you still want history."
* and in "ksh - An Extensible High Level Language" David Korn writes:
"Originally the idea of adding command line editing to ksh was
rejected in the hope that line editing would move into the terminal
driver." [2]
I have always wondered, what such a central terminal driver driven
history/line-editing would have felt like.
[1] https://www.in-ulm.de/~mascheck/various/ash/DIFFERENCES
[2] https://www.in-ulm.de/~mascheck/bourne/korn.html
[SysIII port]
Is there are reason to hang on to the Bourne shell? Maybe shell scripts?
Does it perform better than ksh or bash?
Don't get me wrong, I much prefer the sh syntax over csh syntax, but
I'd never go back to the Bourne shell as my login shell. Way too much
useful stuff in ksh/bash. 
31 Dec
31 Dec
11:40 a.m.
Hi Jon,
Just as history and line editing were a possible enhancement to the TTY
driver, didn't Pike comment that job control was cooked up in foreign
climes because they couldn't drag out a new TTY window in pixels?
Was something like screen(1)'s ‘Ctrl-A c’ considered in the TTY driver
to spin up another pseudo-TTY rather than suspend the long job with
Ctrl-Z and ‘bg’ it?
--
Cheers, Ralph.
I guess in
interactive use most users would only miss one thing:
the history & line editing capability?
Job control?
6:49 p.m.
Ralph Corderoy writes:
Hi Jon,
Just as history and line editing were a possible enhancement to the TTY
driver, didn't Pike comment that job control was cooked up in foreign
climes because they couldn't drag out a new TTY window in pixels?
Was something like screen(1)'s ‘Ctrl-A c’ considered in the TTY driver
to spin up another pseudo-TTY rather than suspend the long job with
Ctrl-Z and ‘bg’ it?
My memory is really fuzzy on this. I recall that job control came along
with csh which I used until bash. At the time there were hardly any
graphics displays used as interactive devices. I'm talking Berkeley job
control here; I seem to remember that someone got wedged into System V
that was awful and unusable.
Jon
I guess
in interactive use most users would only miss one thing:
the history & line editing capability?
Job control?
7:24 p.m.
below
On Sat, Dec 31, 2022 at 1:50 PM Jon Steinhart <jon(a)fourwinds.com> wrote:
My memory is really fuzzy on this. I recall that job
control came along
with csh which I used until bash.
Close. That was the wide distribution of it. Kulp created it originally
for his PDP-11 system in the late 1970s. I'm not sure how, but wnj saw
it and imported the idea to UCB. Joy had stopped hacking on the 11 by
then. So he added the kernel support for Job Control to 4.0 and hacked on
his own shell to add the user interface. This would get widely distributed
as in 4.1BSD and since job control was in csh, and not in sh (and job had
made the root use csh ) BSD users learned to used job control from there;
since 4.1BSD release that spread with wings.
At the time there were hardly any
graphics displays used as interactive devices. I'm talking Berkeley job
control here;
Right - which is Kulp job control to be fair.
I seem to remember that someone got wedged into System
V
that was awful and unusable.
IIRC it was in SVR2 that a version of a job control system added some of
the semantics of Kulp's scheme to the kernel, but not all as you point out
and it was pretty disappointing if you had grown up on BSD systems. Later
POSIX would pick up the Kulp/BSD Job Control definition and by SVR3/SVR4
AT&T fleshed out all all of the semantics.
3 Jan
3 Jan
4:32 p.m.
On 12/31/22 2:24 PM, Clem Cole wrote:
I seem to remember that someone got wedged into
System V
that was awful and unusable.
IIRC it was in SVR2 that a version of a job control system added some of
the semantics of Kulp's scheme to the kernel, but not all as you point out
and it was pretty disappointing if you had grown up on BSD systems. Later
POSIX would pick up the Kulp/BSD Job Control definition and by SVR3/SVR4
AT&T fleshed out all all of the semantics.
There is a pretty good discussion of 4.2 BSD-style job control and ways to
make it fit with System V semantics (mostly controlling terminals and, of
course, process groups) at
https://web.physics.ucsb.edu/~pcs/apps/bash/job.control.ps
This is probably the best description of BSD-style job control, and it is
the inspiration for much of the POSIX job control design (e.g., their
invention of "sessions" as the replacement for System V style process
groups).
Chet
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU chet(a)case.edu http://tiswww.cwru.edu/~chet/
1 Jan
1 Jan
1:51 a.m.
Berkeley Job Control was implemented in the kernel and originally only
supported in CSH. I detested csh, so I figured out how it worked and
hacked it into a 5(r0) Bourne Shell. But he time I did that, all the
other BRL guys were using tcsh and so to counter, I hacked command line
editing into the 5R2 Bourne shell.
------ Original Message ------
From "Jon Steinhart" <jon(a)fourwinds.com>
To tuhs(a)tuhs.org
Date 12/31/2022 1:49:54 PM
Subject [TUHS] Re: A few comments on porting the Bourne shell
Ralph Corderoy writes:
Hi Jon,
Just as history and line editing were a possible enhancement to the TTY
driver, didn't Pike comment that job control was cooked up in foreign
climes because they couldn't drag out a new TTY window in pixels?
Was something like screen(1)'s ‘Ctrl-A c’ considered in the TTY driver
to spin up another pseudo-TTY rather than suspend the long job with
Ctrl-Z and ‘bg’ it?
My memory is really fuzzy on this. I recall that job control came along
with csh which I used until bash. At the time there were hardly any
graphics displays used as interactive devices. I'm talking Berkeley job
control here; I seem to remember that someone got wedged into System V
that was awful and unusable.
Jon I guess
in interactive use most users would only miss one thing:
the history & line editing capability?
Job control? 
2 Jan
2 Jan
8:03 p.m.
New subject: Command Line Editing in the Terminal Driver (Was: A few comments on porting the Bourne shell)
On Fri, Dec 30, 2022, at 12:42, Sven Mascheck wrote:
and in "ksh - An Extensible High Level
Language" David Korn writes:
* "Originally the idea of adding command line editing to ksh was rejected in the
hope that line editing would move into the terminal driver." [2]
[2] https://www.in-ulm.de/~mascheck/bourne/korn.html
This phrase has haunted me for years. It’s so clearly the “correct” separation of
concerns, until you actually attempt implementing it. And Bourne’s irregular grammar
certainly doesn’t help here. I’d prefer to move beyond readline, ideally something like
text objects per vim/zsh/treesitter. But having one parser in the interpreter and another
in the line editor becomes quite exciting if you want a true bourne or even posix sh.
But the thing that brought be back to playing against this quote again this year was
starting to research the QED lineage and discovering helix & kakoune. Honestly, they
feels like the most coherent advancements in QED-style editors since sam & acme. (Yes,
I’m irrationally excluding vim text-objects, mostly because no one removed editor features
that t-o made redundant. It’s as if ed had twice as many commands, one for regexp matches
and one for pre-ken-QEDist exact matches.)
Anyway, the only time I’ve really seen “line editing […] move into the terminal driver”
sound possible was acme’s win. For those who haven’t had the pleasure, rsc describes it at
15:25 in https://research.swtch.com/acme. “I worked for many years in a shell without
history or command line editing, and I never missed it because Acme is providing this for
me.”
It’s hard to convey how surprisingly pleasant sh or rc can be within acme win to someone
who has only used them within a mere terminal-emulator-emulator. Especially since a
greenfield terminal app has more code emulating old xterm behaviors than physical vt100
behaviours. This is especially exhausting when you try to use something like NeoVim’s
:term expecting it to just work and discover that buffer-line-wrapping on window-resize
hasn’t been implemented.
Anyone seen any other “terminal drivers” that provide a pleasant alternative to line
editing?
--
~j
8:33 p.m.
New subject: Command Line Editing in the Terminal Driver
Joseph Holsten wrote:
Anyone seen any other “terminal drivers” that provide
a pleasant
alternative to line editing?
The Stanford AI lab PDP-10 operating system had line editing built in,
and the E editor integrated with this.
11 Jan
11 Jan
2:51 a.m.
New subject: Command Line Editing in the Terminal Driver
On Jan 2, 2023, at 12:33 PM, Lars Brinkhoff <lars(a)nocrew.org> wrote:
Joseph Holsten wrote:
Some operating systems like HP's MPE (and probably also Tandem GUARDIAN) just
expected the use of block-mode terminals for purposes of I/O efficiency since interactive
use was typically via forms-based applications, and got line editing somewhat as a
side-effect.
-- Chris
Anyone seen any other “terminal drivers” that
provide a pleasant
alternative to line editing?
The Stanford AI lab PDP-10 operating system had line editing built in,
and the E editor integrated with this. 
3:53 a.m.
New subject: Command Line Editing in the Terminal Driver
On Tuesday, 10 January 2023 at 18:51:07 -0800, Chris Hanson wrote:
On Jan 2, 2023, at 12:33 PM, Lars Brinkhoff
<lars(a)nocrew.org> wrote:
Guardian was interesting. The normal connection (from my point of
view, anyway) was via a TATM (Tandem Asynchronous Terminal
Multiplexer), which buffered the characters, but it didn't interrupt
the processor until one of 4 configurable characters were entered
(typically CR, abort, end of input). This didn't offer line editing,
and the workarounds were painful.
There was also a block mode, mainly for end user applications, though
the “full screen” editor also used it.
Greg
--
Sent from my desktop computer.
Finger grog(a)lemis.com for PGP public key.
See complete headers for address and phone numbers.
This message is digitally signed. If your Microsoft mail program
reports problems, please read http://lemis.com/broken-MUA.php
The Stanford AI lab PDP-10 operating system had
line editing built in,
and the E editor integrated with this.
Some operating systems like HP's MPE (and probably also Tandem
GUARDIAN) just expected the use of block-mode terminals for purposes
of I/O efficiency since interactive use was typically via
forms-based applications, and got line editing somewhat as a
side-effect. 
2 Jan
2 Jan
9:08 p.m.
New subject: Command Line Editing in the Terminal Driver (Was: A few comments on porting the Bourne shell)
It just seems much more natural to me to have line editing provided at some
fundamental level. Putting it in the shell or some library means some tools
get it but some don't. Why should sh have editing but mail not? But then
there's glob.
Do it once, do it well, and do it for everything, where "it" is a wildcard
that refers to every important detail.
-rob
On Tue, Jan 3, 2023 at 7:03 AM Joseph Holsten <joseph(a)josephholsten.com>
wrote:
On Fri, Dec 30, 2022, at 12:42, Sven Mascheck wrote:
and in "ksh - An Extensible High Level Language" David Korn writes:
- "Originally the idea of adding command line editing to ksh was
rejected in the hope that line editing would move into the terminal
driver." [2]
[2] https://www.in-ulm.de/~mascheck/bourne/korn.html
This phrase has haunted me for years. It’s so clearly the “correct”
separation of concerns, until you actually attempt implementing it. And
Bourne’s irregular grammar certainly doesn’t help here. I’d prefer to move
beyond readline, ideally something like text objects per
vim/zsh/treesitter. But having one parser in the interpreter and another in
the line editor becomes quite exciting if you want a true bourne or even
posix sh.
But the thing that brought be back to playing against this quote again
this year was starting to research the QED lineage and discovering helix &
kakoune. Honestly, they feels like the most coherent advancements in
QED-style editors since sam & acme. (Yes, I’m irrationally excluding vim
text-objects, mostly because no one removed editor features that t-o made
redundant. It’s as if ed had twice as many commands, one for regexp matches
and one for pre-ken-QEDist exact matches.)
Anyway, the only time I’ve really seen “line editing […] move into the
terminal driver” sound possible was acme’s win. For those who haven’t had
the pleasure, rsc describes it at 15:25 in https://research.swtch.com/acme.
“I worked for many years in a shell without history or command line
editing, and I never missed it because Acme is providing this for me.”
It’s hard to convey how surprisingly pleasant sh or rc can be within acme
win to someone who has only used them within a mere
terminal-emulator-emulator. Especially since a greenfield terminal app
has more code emulating old xterm behaviors than physical vt100 behaviours.
This is especially exhausting when you try to use something like NeoVim’s
:term expecting it to just work and discover that buffer-line-wrapping on
window-resize hasn’t been implemented.
Anyone seen any other “terminal drivers” that provide a pleasant
alternative to line editing?
--
~j
3 Jan
3 Jan
4:53 p.m.
New subject: Command Line Editing in the Terminal Driver (Was: A few comments on porting the Bourne shell)
Along these lines but not quite, Jupyter Notebooks have stood out to me as
another approach to this concept, with behavior I'd like to see implemented
in a shell.
With these notebooks, users type text into blocks, and then can have these
blocks evaluated by some underlying engine, with the results of the
evaluation showing up in a block underneath where they entered text.
Results can be not only text, but things like rendered charts and images.
There may be GUI-interactable rendered elements as well, but I'm unsure.
Notably, text-entry blocks are created such that each new block holds onto
the context of the previous blocks. So, a variable defined in the first
eval block will be available in later blocks.
Currently these are used primarily for demonstrating APIs, exploring data
with python, or writing quick PoCs that can then be extracted into an
application. Some companies, such as Netflix, have experimented with using
them entirely as a replacement for shell scripts, which is the sort of
research I'd most love to see.
Fundamentally, I think it's a smell that the way we interact with our
systems is tied to the long-gone hardware requirement of "be usable with a
teletype." Acme was certainly a move towards recognizing the value of
being able to reuse space that, in TTYs, would be 'dead' text. The ability
to evaluate commands in Acme, and the further inclusion of an underlying
evaluator that holds state by Jupyter were good steps forward, I think.
I'd love to see experimentation with a whole system that takes the jupyter
approach, with nested namespaces forming applications, combined with data
being held in "blocks" as well as code - much like acme opens and edits
files as well as letting you execute either them or snippets in them. I
think there's a chance something could be developed that would better fit
the way we interface with computers today, and the underlying engine
approach would move us toward the "everything speaks one language" design
we lost in the move from shells to standalone GUI applications.
Mostly, I'd be bummed to see yet another shell that pretends it's behind a
"glass teletype", to use the tongue-in-cheek term from "Ed Mastery",
or
just continuing elaboration on that design, despite the fact I use shells
daily. It's the best we have now, but I feel like it's the way forward.
Cheers!
Marshall
P.S. - Kakoune is neat and I'm going to have to poke around at that, thanks
for pointing it out.
On Mon, Jan 2, 2023 at 3:03 PM Joseph Holsten <joseph(a)josephholsten.com>
wrote:
On Fri, Dec 30, 2022, at 12:42, Sven Mascheck wrote:
and in "ksh - An Extensible High Level Language" David Korn writes:
- "Originally the idea of adding command line editing to ksh was
rejected in the hope that line editing would move into the terminal
driver." [2]
[2] https://www.in-ulm.de/~mascheck/bourne/korn.html
This phrase has haunted me for years. It’s so clearly the “correct”
separation of concerns, until you actually attempt implementing it. And
Bourne’s irregular grammar certainly doesn’t help here. I’d prefer to move
beyond readline, ideally something like text objects per
vim/zsh/treesitter. But having one parser in the interpreter and another in
the line editor becomes quite exciting if you want a true bourne or even
posix sh.
But the thing that brought be back to playing against this quote again
this year was starting to research the QED lineage and discovering helix &
kakoune. Honestly, they feels like the most coherent advancements in
QED-style editors since sam & acme. (Yes, I’m irrationally excluding vim
text-objects, mostly because no one removed editor features that t-o made
redundant. It’s as if ed had twice as many commands, one for regexp matches
and one for pre-ken-QEDist exact matches.)
Anyway, the only time I’ve really seen “line editing […] move into the
terminal driver” sound possible was acme’s win. For those who haven’t had
the pleasure, rsc describes it at 15:25 in https://research.swtch.com/acme.
“I worked for many years in a shell without history or command line
editing, and I never missed it because Acme is providing this for me.”
It’s hard to convey how surprisingly pleasant sh or rc can be within acme
win to someone who has only used them within a mere
terminal-emulator-emulator. Especially since a greenfield terminal app
has more code emulating old xterm behaviors than physical vt100 behaviours.
This is especially exhausting when you try to use something like NeoVim’s
:term expecting it to just work and discover that buffer-line-wrapping on
window-resize hasn’t been implemented.
Anyone seen any other “terminal drivers” that provide a pleasant
alternative to line editing?
--
~j
4 Jan
4 Jan
9:18 a.m.
New subject: Command Line Editing in the Terminal Driver (Was: A few comments on porting the Bourne shell)
On Jan 3, 2023, at 08:55, Marshall Conover
<marzhall.o(a)gmail.com> wrote:
Along these lines but not quite, Jupyter Notebooks have stood out to me as another
approach to this concept, with behavior I'd like to see implemented in a shell.
…
Currently these are used primarily for demonstrating
APIs, exploring data with python, or writing quick PoCs that can then be extracted into an
application. Some companies, such as Netflix, have experimented with using them entirely
as a replacement for shell scripts, which is the sort of research I'd most love to
see.
…
I'd love to see experimentation with a whole
system that takes the jupyter approach, with nested namespaces forming applications,
combined with data being held in "blocks" as well as code - much like acme opens
and edits files as well as letting you execute either them or snippets in them. I think
there's a chance something could be developed that would better fit the way we
interface with computers today, and the underlying engine approach would move us toward
the "everything speaks one language" design we lost in the move from shells to
standalone GUI applications.
This sounds tremendously useful. It gets at that feeling I want to have from emacs and
literate programming but rarely achieve, the sense of Living Text.
11 Jan
11 Jan
2:56 a.m.
New subject: Command Line Editing in the Terminal Driver (Was: A few comments on porting the Bourne shell)
On Jan 3, 2023, at 8:53 AM, Marshall Conover <marzhall.o(a)gmail.com> wrote:
With these notebooks, users type text into blocks, and then can have these blocks
evaluated by some underlying engine, with the results of the evaluation showing up in a
block underneath where they entered text. Results can be not only text, but things like
rendered charts and images. There may be GUI-interactable rendered elements as well, but
I'm unsure.
This is basically the model created by Symbolics for Dynamic Windows, which was the basis
for CLIM, and which I believe they also adopted in the Symbolics distribution of MACSYMA.
Kalman Reti does a great job demonstrating it under OpenGenera here:
https://www.youtube.com/watch?v=o4-YnLpLgtk
(Jupyter notebooks are a derivative of Mathematica notebooks, which bear more than a
passing resemblance to the Dynamic Windows stuff just like Mathematica bears more than a
passing resemblance to McCarthy's "M-expressions" and the MACSYMA system…)
-- Chris
30 Dec
30 Dec
8:47 p.m.
On 12/30/22 3:02 PM, Larry McVoy wrote:
On Fri, Dec 30, 2022 at 02:51:26PM -0500, Chet Ramey
wrote:
Historical interest? Software archaeology? Reference behavior?
I don't think anyone is suggesting that we use it as a login shell, in the
same way that no one is suggesting we go back to using v7 as an everyday
computing environment. The world's come too far.
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU chet(a)case.edu http://tiswww.cwru.edu/~chet/
On 12/30/22 1:25 PM, Paul Ruizendaal wrote:
Is there are reason to hang on to the Bourne shell? Maybe shell scripts?
Does it perform better than ksh or bash?
London and Reiser report about porting the shell that ???it required by far the largest
conversion effort of any supposedly portable program, for the simple reason that it is not
portable.??? By the time of SysIII this is greatly improved, but also in porting the
SysIII user land it was the most complex of the set so far.
Have you read
http://www.collyer.net/who/geoff/sh.tour.pdf
and looked at http://www.collyer.net/who/geoff/v7sh.tar ?
In the limited literature on Bourne Shell porting, this is authoritative. 
31 Dec
31 Dec
12:08 a.m.
The Almquist shell (ash) and Debian's version of it, "dash", are
compatible re-implementations of the Bourne shell, and use much less
resources, and therefore wind up being faster as processes come and go.
I use csh (tcsh) and bash at the command line, but straight Bourne shell
language is preferred and recommended by many for shell programming, I
used to use csh for that and got bitten by all the things that I later
discovered, those in the know had been warning about for years. Also,
"bash-isms", syntactic sugary things in bash had led me to use them as a
crutch, my scripts got simpler and more to the point when I re-wrote
them for Bourne shell language only. That was my experience. I think
we'll always have some kind of Bourne shell as the script workhorse, at
last in Linux/Unix start-up and other blood and guts stuff.
In John Gilmore's paper on porting BSD through gcc to get to 4.4, he
also remarked on how the original Bourne shell code was difficult to port.
On 12/30/2022 01:02 PM, Larry McVoy wrote:
On Fri, Dec 30, 2022 at 02:51:26PM -0500, Chet Ramey
wrote:
On 12/30/22 1:25 PM, Paul Ruizendaal wrote:
Is there
are reason to hang on to the Bourne shell? Maybe shell scripts?
Does it perform better than ksh or bash?
Don't get me wrong, I much prefer the sh syntax over csh syntax, but
I'd never go back to the Bourne shell as my login shell. Way too much
useful stuff in ksh/bash.
London and Reiser report about porting the shell
that ???it required by far the largest conversion effort of any supposedly portable
program, for the simple reason that it is not portable.??? By the time of SysIII this is
greatly improved, but also in porting the SysIII user land it was the most complex of the
set so far.
Have you read
http://www.collyer.net/who/geoff/sh.tour.pdf
and looked at http://www.collyer.net/who/geoff/v7sh.tar ?
In the limited literature on Bourne Shell porting, this is authoritative.
3:59 a.m.
On Fri, Dec 30, 2022 at 05:08:42PM -0700, Luther Johnson wrote:
I use csh (tcsh) and bash at the command line, but
straight Bourne shell
language is preferred and recommended by many for shell programming, I used
to use csh for that and got bitten by all the things that I later
discovered, those in the know had been warning about for years. Also,
"bash-isms", syntactic sugary things in bash had led me to use them as a
crutch, my scripts got simpler and more to the point when I re-wrote them
for Bourne shell language only. That was my experience. I think we'll always
have some kind of Bourne shell as the script workhorse, at last in
Linux/Unix start-up and other blood and guts stuff.
When I was running my engineering team I was strict about Bourne syntax
and features only. I got pushed on like crazy because "bash has this
$GOODNESS whhhhhhhy can't we use it". Because we were supporting our
product on pretty much every unix and if it wasn't HP-UX that had an
ancient /bin/sh, it was AIX or whoever.
Over and over, I won the "straight bourne shell only" battle. So I agree,
if you want /bin/sh to work, Bourne shell for the win.
For a login shell, bash is my shell of choice. It's bloated but I'm
typing this on a 5 year old Lenova X1 Carbon with 16GB of memory and
4 cores and it's fine. It was fine a 133mhz Pentium.
4:12 a.m.
On Fri, 30 Dec 2022, Larry McVoy wrote:
When I was running my engineering team I was strict
about Bourne syntax
and features only. I got pushed on like crazy because "bash has this
$GOODNESS whhhhhhhy can't we use it". Because we were supporting our
product on pretty much every unix and if it wasn't HP-UX that had an
ancient /bin/sh, it was AIX or whoever.
Over and over, I won the "straight bourne shell only" battle. So I agree,
if you want /bin/sh to work, Bourne shell for the win.
For a login shell, bash is my shell of choice. It's bloated but I'm
typing this on a 5 year old Lenova X1 Carbon with 16GB of memory and
4 cores and it's fine. It was fine a 133mhz Pentium.
There's some variants of the Almquist shell that have bash-style
command-line editing, and I can deal with that if that's /bin/sh. Usually
I use bash and it's plenty fine.
I generally code shell scripts for Posix sh as my baseline and start my
scripts with #!/bin/sh - but sometimes I'll use #!/bin/ksh if I don't
expect them to be used on another machine. ksh93 is lighter and faster
than bash and the only things I miss are quirks of the command line
editor.
-uso.
4:18 a.m.
On Dec 30, 2022, at 7:59 PM, Larry McVoy <lm(a)mcvoy.com> wrote:
For a login shell, bash is my shell of choice. It's bloated but I'm
typing this on a 5 year old Lenova X1 Carbon with 16GB of memory and
4 cores and it's fine. It was fine a 133mhz Pentium.
I switched to zsh in 1993-94 when I found out it had sh compatible
syntax but also had some useful features from csh (bang commands,
a{b,c} expansion, aliases etc.). Back then my machine had 16MB of
memory (IIRC a 50Mhz i486). zsh is the Cadillac of shells. But I
also use rc, the VWbug of shells on plan9!
4:40 a.m.
On Fri, Dec 30, 2022 at 08:18:06PM -0800, Bakul Shah wrote:
On Dec 30, 2022, at 7:59 PM, Larry McVoy
<lm(a)mcvoy.com> wrote:
This should probably move to COFF (which unlike many other lists I've
been on, has critical mass in the "go over there" list).
You are the 2nd to say zsh is your login shell. I tried it for a while,
my progression was ${I dunno} to csh, actually used Bourne shell as a
login shell for a while around 1985 or so, all of my login scripts, other
shell scripts, were Bourne shell, I think zsh was next, then sighed in
relief with the original ksh release that went out, and when bash came
around with portability to everywhere, it has been bash. ksh93 was
interesting but bash was good enough and clearly open source so...
For script portability, pure Bourne shell has been the win for at least
40 years. I suspect my kids and their kids will be writing Steve's
syntax, that's a legacy.
For general purpose scripting, I've moved on. I loved Perl4. It's a
kitchen sink language but it so clearly had roots in the Bourne shell,
in awk, in sed. It was trying to be all of those things in one language
and it mostly succeeded. In the 40mhz SPARC days I actually advocated
for rewriting a ton of /usr/bin on SunOS in perl4 because it was so easy
and far more maintainable than the C versions.
These days perl is weird, the kids seem to like Python, I hate Python
because there is no printf in the base language. How is that a thing?
But people like it.
--
---
Larry McVoy Retired to fishing http://www.mcvoy.com/lm/boat
For a login shell, bash is my shell of choice. It's bloated but I'm
typing this on a 5 year old Lenova X1 Carbon with 16GB of memory and
4 cores and it's fine. It was fine a 133mhz Pentium.
I switched to zsh in 1993-94 when I found out it had sh compatible
syntax but also had some useful features from csh (bang commands,
a{b,c} expansion, aliases etc.). 
4:19 a.m.
Fascinating.
When I left IBM Research to become a grad student at CMU (1981) the Unix CS
was running did not have history in the shell. It had just been introduced
in VM/CMS and I loved it. I nagged the sysadmins in the CS department
about it, and voila, it appeared shortly thereafter. I presume it was one
of the things discussed here ported to the CMU environment.
I remember that I implemented history for the shell in the PERQ, doing some
nasty stuff to fit a reasonable history length in the weird Pascal they
had, since Pascal didn’t have dynamic strings or real pointers.
Marc
=====
On Fri, Dec 30, 2022 at 10:59 PM Larry McVoy <lm(a)mcvoy.com> wrote:
On Fri, Dec 30, 2022 at 05:08:42PM -0700, Luther
Johnson wrote:
--
=====
nygeek.net
mindthegapdialogs.com/home <https://www.mindthegapdialogs.com/home>
I use csh (tcsh) and bash at the command line,
but straight Bourne shell
language is preferred and recommended by many for shell programming, I
used
to use csh for that and got bitten by all the
things that I later
discovered, those in the know had been warning about for years. Also,
"bash-isms", syntactic sugary things in bash had led me to use them as a
crutch, my scripts got simpler and more to the point when I re-wrote them
for Bourne shell language only. That was my experience. I think we'll
always
have some kind of Bourne shell as the script
workhorse, at last in
Linux/Unix start-up and other blood and guts stuff.
When I was running my engineering team I was strict about Bourne syntax
and features only. I got pushed on like crazy because "bash has this
$GOODNESS whhhhhhhy can't we use it". Because we were supporting our
product on pretty much every unix and if it wasn't HP-UX that had an
ancient /bin/sh, it was AIX or whoever.
Over and over, I won the "straight bourne shell only" battle. So I agree,
if you want /bin/sh to work, Bourne shell for the win.
For a login shell, bash is my shell of choice. It's bloated but I'm
typing this on a 5 year old Lenova X1 Carbon with 16GB of memory and
4 cores and it's fine. It was fine a 133mhz Pentium.
4:23 a.m.
On Fri, 30 Dec 2022, Larry McVoy wrote:
When I was running my engineering team I was strict
about Bourne syntax
and features only. I got pushed on like crazy because "bash has this
$GOODNESS whhhhhhhy can't we use it". Because we were supporting our
product on pretty much every unix and if it wasn't HP-UX that had an
ancient /bin/sh, it was AIX or whoever.
I've never bothered to learn those Bash thingies, because "expr" does
everything that I need and is available on just about all boxes.
Over and over, I won the "straight bourne shell
only" battle. So I
agree, if you want /bin/sh to work, Bourne shell for the win.
Yep; whoever wrote CSH must've been high on something, as the syntax makes
no sense whatsoever.
For a login shell, bash is my shell of choice.
It's bloated but I'm
typing this on a 5 year old Lenova X1 Carbon with 16GB of memory and 4
cores and it's fine. It was fine a 133mhz Pentium.
I do admit to being a bit of a ZSH user... I've never bothered to learn
all its features, but the subset I use gets me through.
-- Dave
4:37 a.m.
Bourne to program, type with Joy.
On Fri, Dec 30, 2022 at 11:23 PM Dave Horsfall <dave(a)horsfall.org> wrote:
On Fri, 30 Dec 2022, Larry McVoy wrote:
--
Sent from a handheld expect more typos than usual
When I was running my engineering team I was
strict about Bourne syntax
and features only. I got pushed on like crazy because "bash has this
$GOODNESS whhhhhhhy can't we use it". Because we were supporting our
product on pretty much every unix and if it wasn't HP-UX that had an
ancient /bin/sh, it was AIX or whoever.
I've never bothered to learn those Bash thingies, because "expr" does
everything that I need and is available on just about all boxes.
Over and over, I won the "straight bourne
shell only" battle. So I
agree, if you want /bin/sh to work, Bourne shell for the win.
Yep; whoever wrote CSH must've been high on something, as the syntax makes
no sense whatsoever.
For a login shell, bash is my shell of choice.
It's bloated but I'm
typing this on a 5 year old Lenova X1 Carbon with 16GB of memory and 4
cores and it's fine. It was fine a 133mhz Pentium.
I do admit to being a bit of a ZSH user... I've never bothered to learn
all its features, but the subset I use gets me through.
-- Dave
2 Jan
2 Jan
5:10 a.m.
On 12/30/22 20:23, Dave Horsfall wrote:
Yep; whoever wrote CSH must've been high on
something, as the syntax makes
no sense whatsoever.
Bill Joy wrote csh. He based it on the V6 shell. It made perfect sense
at the time. This was before the V7 Bourne shell came out.
Thanks,
/Mary Ann Horton/ (she/her/ma'am)
maryannhorton.com <https://maryannhorton.com>
"This is a great book" - Monica Helms
"Brave and Important" - Laura L. Engel
Available on Amazon and bn.com!
<https://www.amazon.com/Trailblazer-Lighting-Transgender-Equality-Corporate-ebook/dp/B0B8F2BR9B>
4:25 p.m.
Be careful in applying modern judgments without remembering
the historical context - below
On Mon, Jan 2, 2023 at 12:12 AM Mary Ann Horton <mah(a)mhorton.net> wrote:
On 12/30/22 20:23, Dave Horsfall wrote:
Yep; whoever wrote CSH must've been high on something, as the syntax makes
no sense whatsoever.
Bill Joy wrote csh. He based it on the V6 shell. It made perfect sense at
the time. This was before the V7 Bourne shell came out.
Right -- let me add a little detail to this that might help. With the
Fourth, Fifth, and Sixth Edition the default shell from Research is the
Thompson shell. It's simple and un-adorn. Get the job done for typing,
but as a formal scripting (programming) language is not easy to use.
In fact, the command 'cd' idiom is still 'chdir' [add it was a
typical
local hack that many of did]. If you look at the USENIX tapes in the TUHS
archives, in particular, the Harvard tape I suspect you will find a number
of alternative shells that people had offered up from the different sites.
With 'BSD' (the first UNIX distribution tape from the UCB Industrial
Liasion Office of EECS which we often refer to on this list as 1BSD to
distinguish it from the other releases - also available on TUHS], there is
a new Berkeley shell, that Mary Ann points out Bill had created by
modifying the Thompson shell and added some features to it. By the time of
the 2BSD, the shell has been further been refined and by then renamed the
csh, as the syntax had been made to approximate the C language.
A big difference between the *Thompson Shell begets -> Berkeley Shell ->
begets C Shell *and Bourne's work on his shell, is that Steve started from
scratch. Steve had just come off many years of the Algol 68 project and was
certainly extremely well-versed in the detailed subtleties of the semantics
of language design. His higher bit was to create something *easier for
people to write scripts using the -> small is beautiful -> Unix philosophy
ideas*.
Bill, on the other hand, is the quintessential hacker (My line is him in
those days used to be: "He types open curly brace, close curly brace and he
patches at 9600 baud.") Bill was interested in a shell that made typing
easy, was not a load the system, and made it easy for >>him<< to write
simple admin scripts.
I would suggest both were 100% successful at what their purpose was.
Like Larry, I have always been a meany and I have tried to make darned sure
my folks (even to this day - we will see how long that last after I retire)
use V7 syntax scripts - not bash, not even korn. As I said, the old UNIX
hacker wisdom, has been "*Bourne to Program, Type with **Joy*" and you have
a hard time going wrong.
[FYI - while I have tcsh around, on my Mac these days I finally switched
from tcsh to zsh as my default, but pretty much all of Bill's typing tricks
are supported in zsh so the 'ROMs" in my fingers are happy after >40
years"] but all of my scripts are still V7 syntax - never been a reason to
change that -- bigger issues usually found are GNU/Linux is not UNIX and
Apple had to pee on things also.
ᐧ
4:51 p.m.
On Mon, Jan 02, 2023 at 11:25:06AM -0500, Clem Cole wrote:
Like Larry, I have always been a meany and I have
tried to make darned sure
my folks (even to this day - we will see how long that last after I retire)
use V7 syntax scripts - not bash, not even korn.
I think it is less of an issue today but if I were still supporting a
multi platform product, I'd still insist on it. Ya never know when a
big pile of money is going to show and insist that you work on SCO
based cash registers or something. Every. Single. Time. one of
my engineers whined that they liked bash better, we'd hit on some
supported platform where /bin/sh is not bash. And as we shipped
scripts that people expected to work, you can't fix their /bin/sh.
I won that argument over and over and eventually we all instinctively
wrote portable scripts.
5:32 p.m.
On Jan 2, 2023, at 9:51 AM, Larry McVoy
<lm(a)mcvoy.com> wrote:
On Mon, Jan 02, 2023 at 11:25:06AM -0500, Clem Cole wrote:
I feel like after POSIX.2 had had a few years to settle, it was OK to use *that* as your
scripting language (I particularly like $() rather than backticks, because of nestability,
although you can certainly argue that if you're regularly nesting your
interpolations, it's the design that's broken, not the specific syntax of the
command).
At least as long as you put a very visible comment right at the top saying
"We're assuming POSIX shell."
But of course the "after it had had a few years to settle" is doing a lot of
lifting there. During the Unix Wars, yeah, clearly using v7 sh as
lowest-common-denominator was the right choice. After the establishment of Linux Hegemony
sometime in the early 2000s, I think I remember that you could pretty much count on
everyone at least providing a POSIX shell, even if it wasn't the default. (But of
course by then, "everyone" that wasn't Linux or OS X was already down to
more-or-less Sun, IBM, HP, and DEC/Compaq.)
These days, if you have a vendor-supplied shell script that DOESN'T work correctly
under a POSIX /bin/sh, that should be treated as a defect.
Adam
Like Larry, I have always been a meany and I have
tried to make darned sure
my folks (even to this day - we will see how long that last after I retire)
use V7 syntax scripts - not bash, not even korn.
And as we shipped
scripts that people expected to work, you can't fix their /bin/sh.
I won that argument over and over and eventually we all instinctively
wrote portable scripts.
5:43 p.m.
On Mon, Jan 02, 2023 at 10:32:50AM -0700, Adam Thornton wrote:
But of course the "after it had had a few years
to settle" is doing
a lot of lifting there. During the Unix Wars, yeah, clearly using
v7 sh as lowest-common-denominator was the right choice. After the
establishment of Linux Hegemony sometime in the early 2000s, I think I
remember that you could pretty much count on everyone at least providing
a POSIX shell, even if it wasn't the default. (But of course by then,
"everyone" that wasn't Linux or OS X was already down to more-or-less
Sun,
IBM, HP, and DEC/Compaq.)
I was supporting a commercial product in the early 2000's and there
were all sorts of systems then that had old shells. Yes, you could make
everything a 2 level thing where the first level finds the correct shell,
but that's just fuss. Just make things portable, it's not that hard
and it works everywhere.
When you get to the commercial world, you'd be stunned to see how long
old machines last. If they are solving some problem, and they aren't
broken, nobody replaces them. I'll bet you anything there are still
SCO registers out there, I'll bet there are still PDP-11s out there.
If it ain't broke, don't fix it.
5:48 p.m.
Arnold Schoenberg said of his 12-tone method of music composition, "in
constraints I find freedom". The best thing about committing to a simple
style and a small set of tools that span the problem space, is the
effect it has on my program composition, imho.
On 01/02/2023 10:43 AM, Larry McVoy wrote:
On Mon, Jan 02, 2023 at 10:32:50AM -0700, Adam
Thornton wrote:
But of course the "after it had had a few
years to settle" is doing
a lot of lifting there. During the Unix Wars, yeah, clearly using
v7 sh as lowest-common-denominator was the right choice. After the
establishment of Linux Hegemony sometime in the early 2000s, I think I
remember that you could pretty much count on everyone at least providing
a POSIX shell, even if it wasn't the default. (But of course by then,
"everyone" that wasn't Linux or OS X was already down to more-or-less
Sun,
IBM, HP, and DEC/Compaq.)
I was supporting a commercial product in the early
2000's and there
were all sorts of systems then that had old shells. Yes, you could make
everything a 2 level thing where the first level finds the correct shell,
but that's just fuss. Just make things portable, it's not that hard
and it works everywhere.
When you get to the commercial world, you'd be stunned to see how long
old machines last. If they are solving some problem, and they aren't
broken, nobody replaces them. I'll bet you anything there are still
SCO registers out there, I'll bet there are still PDP-11s out there.
If it ain't broke, don't fix it.
6 p.m.
At 2023-01-02T10:48:32-0700, Luther Johnson wrote:
Arnold Schoenberg said of his 12-tone method of music
composition, "in
constraints I find freedom". The best thing about committing to a
simple style and a small set of tools that span the problem space, is
the effect it has on my program composition, imho.
Schoenberg didn't have to compose for orchestras such that the Berlin
Philharmonic string section tuned half of the octave meantone with the
other half equal-tempered, whereas the Royal Concertgebouw had the
brasses play an un-notated ascending major third gliss anytime you
approached the tonic from a tritone _below_ (only).
On 01/02/2023 10:43 AM, Larry McVoy wrote:
> I was supporting a commercial product in the early 2000's and there
> were all sorts of systems then that had old shells. Yes, you could
> make everything a 2 level thing where the first level finds the
> correct shell, but that's just fuss. Just make things portable,
> it's not that hard and it works everywhere.
>
> When you get to the commercial world, you'd be stunned to see how
> long old machines last. If they are solving some problem, and they
> aren't broken, nobody replaces them. I'll bet you anything there
> are still SCO registers out there, I'll bet there are still PDP-11s
> out there.
The logical consequence of this is to write in Autoconf's recommended
dialect of shell, which has far too many rules to remember because every
Unix vendor added exciting new bugs to their "pure" V7 shells.
https://www.gnu.org/software/autoconf/manual/autoconf-2.60/html_node/Portab…
> If it ain't broke, don't fix it.
They were all broken. Badly. If your scripts worked, you got lucky at
Russian Roulette. No greater claim to robustness can be made.
POSIX shell conformance still proves challenging for vendors, but is an
immense improvement over the status quo ante.
I agree with Robert Elz though that standardizing the alias feature was
nuts. I want to know who brought the crack cocaine to the conference
table that day.
Regards,
Branden
6:05 p.m.
Point taken, if not entirely understood :)
On 01/02/2023 11:00 AM, G. Branden Robinson wrote:
At 2023-01-02T10:48:32-0700, Luther Johnson wrote:
Arnold Schoenberg said of his 12-tone method of
music composition, "in
constraints I find freedom". The best thing about committing to a
simple style and a small set of tools that span the problem space, is
the effect it has on my program composition, imho.
Schoenberg didn't have to
compose for orchestras such that the Berlin
Philharmonic string section tuned half of the octave meantone with the
other half equal-tempered, whereas the Royal Concertgebouw had the
brasses play an un-notated ascending major third gliss anytime you
approached the tonic from a tritone _below_ (only).
On 01/02/2023 10:43 AM, Larry McVoy wrote:
> I was supporting a commercial product in the early 2000's and there
> were all sorts of systems then that had old shells. Yes, you could
> make everything a 2 level thing where the first level finds the
> correct shell, but that's just fuss. Just make things portable,
> it's not that hard and it works everywhere.
>
> When you get to the commercial world, you'd be stunned to see how
> long old machines last. If they are solving some problem, and they
> aren't broken, nobody replaces them. I'll bet you anything there
> are still SCO registers out there, I'll bet there are still PDP-11s
> out there.
The logical consequence of this is to write in Autoconf's
recommended
dialect of shell, which has far too many rules to remember because every
Unix vendor added exciting new bugs to their "pure" V7 shells.
https://www.gnu.org/software/autoconf/manual/autoconf-2.60/html_node/Portab…
> If it ain't broke, don't fix it.
They were all broken. Badly. If your scripts worked, you got lucky at
Russian Roulette. No greater claim to robustness can be made.
POSIX shell conformance still proves challenging for vendors, but is an
immense improvement over the status quo ante.
I agree with Robert Elz though that standardizing the alias feature was
nuts. I want to know who brought the crack cocaine to the conference
table that day.
Regards,
Branden
6:12 p.m.
On Mon, Jan 02, 2023 at 12:00:20PM -0600, G. Branden Robinson wrote:
You are talking to a dude with 40+ years of Unix experience, supporting
commercial products most of that time. I didn't get "lucky at Russian
Roulette", I wrote scripts that were portable. I have 40 year old scripts
that _still_ work and they work on virtually every Unix ever built.
How do I know? I was a contractor for my first job, I got plopped down
in front of every random unix you could imagine and each time I polished
off the warts.
I spent decades supporting my own products on every flavor of Unix and
processors from Arm to System/360. Oh, and Windows XP and on and MacOS.
My scripts worked with /bin/sh being whatever it was.
It's interesting to me that other old timers, like Clem, are saying
exactly the same thing as I am. Are we all wrong?
> If it
ain't broke, don't fix it.
They were all broken. Badly. If your scripts worked, you got lucky at
Russian Roulette. No greater claim to robustness can be made.
POSIX shell conformance still proves challenging for vendors, but is an
immense improvement over the status quo ante.
6:16 p.m.
On Mon, Jan 2, 2023 at 1:13 PM Larry McVoy <lm(a)mcvoy.com> wrote:
My scripts worked with /bin/sh being whatever it was.
It's interesting to me that other old timers, like Clem, are saying
exactly the same thing as I am. Are we all wrong?
Exactly -- if you are disciplined -- it just works. It's not magic.
ᐧ
7:50 p.m.
On Mon, Jan 2, 2023, 11:18 AM Clem Cole <clemc(a)ccc.com> wrote:
On Mon, Jan 2, 2023 at 1:13 PM Larry McVoy <lm(a)mcvoy.com> wrote:
I've only got 30 years experience in writing portable shell scripts... But
I have a slightly different take. There is an element of luck involved.
There are a number of constructs that are valid in v7 shell that aren't
completely portable due to bugs in vendor shells. The luck part comes in
either you know what to avoid because you got unlucky and got burned in the
past. Or you got lucky and never thought to write something like that so it
just works. With experience, you know what works...
I notice the old timers hedged a little bit by saying things like
"virtually all" so maybe they had a bit of bad luck now and again...
Also, the really aweful systems are rare enough today that most folks on
this list have had the good luck not to need to run on them... :)
You can get super close though, but I'll bet there's some scripts that work
only because they didn't run on, say, Eunice...
But luck favors the prepared... and making the effort to try to be portable
will help when you have to run on dash, or one of the BSD shells that might
not have $<(cmd) or some other useful but not universal feature...
Warner
My scripts worked with /bin/sh being whatever it was.
It's interesting to me that other old timers, like Clem, are saying
exactly the same thing as I am. Are we all wrong?
Exactly -- if you are disciplined -- it just works. It's not magic.
ᐧ
8:05 p.m.
On Jan 2, 2023, at 12:50 PM, Warner Losh
<imp(a)bsdimp.com> wrote:
You can get super close though, but I'll bet there's some scripts that work
only because they didn't run on, say, Eunice...
"Congratulations!"
But luck favors the prepared... and making the effort to try to be portable will help
when you have to run on dash, or one of the BSD shells that might not have $<(cmd) or
some other useful but not universal feature...
Yeah and I think we're basically agreeing here. The problem isn't *really*
whether you use some-odd-number-of-backslashes-and-a-backtick versus dollar-open-paren,
the problem is people who put #!/bin/sh at the top of the script when the contents only
run under bash. And there's a lot of that, because just as all the world was VAX
once upon a time, all the world is assumed to be an x86_64 Linux where sh is bash.
At least the move towards containerization has largely defeated that assumption now,
although in some sense it's also just swept the problem under someone else's
rug.
Adam
7:21 p.m.
At 2023-01-02T10:12:42-0800, Larry McVoy wrote:
You are talking to a dude with 40+ years of Unix
experience,
I'm clocking 30 this year. A babe in the woods.
supporting commercial products most of that time. I
didn't get "lucky
at Russian Roulette", I wrote scripts that were portable. I have 40
year old scripts that _still_ work and they work on virtually every
Unix ever built. How do I know? I was a contractor for my first job,
I got plopped down in front of every random unix you could imagine and
each time I polished off the warts.
Yes. That's how the chapter of the GNU Autoconf manual to which I
linked got written--or so I surmise, based simply on how bizarre some of
the bugs are that have to be worked around.
I spent decades supporting my own products on every
flavor of Unix and
processors from Arm to System/360. Oh, and Windows XP and on and
MacOS.
My scripts worked with /bin/sh being whatever it was.
It's interesting to me that other old timers, like Clem, are saying
exactly the same thing as I am. Are we all wrong?
Over on the groff list you had the opposite advice regarding Make: pick
one implementation and go with it, you said, and other implementations
be damned (very loosely paraphrasing).[1]
I'm not saying you're wrong--I'm saying you're inconsistent.
I reckon you did what you needed to do to keep the lights on and the
engineers fed. That's fine, but it's also a recipe for passivity. You
don't have to tell me, but count up the number of times you flexed to
accommodate a buggy implementation versus telling your client that their
shell was just too crap and that they needed to either install one in
which you had confidence, or you'd have to decline the contract.
If their users had demanded more of their commercial Unix vendors, maybe
those Unices would not be regarded as dinosaurs today.
I could harp further on quality of implementation issues, but I'd
probably do better to take up the lotus position on the back of a
motorcycle.
Preferably not while the vehicle is in motion.
Regards,
Branden
[1] https://lists.gnu.org/archive/html/groff/2022-03/msg00084.html
7:34 p.m.
The difference is that "make" need only run on your build systems.
Installation scripts need to run on customer systems.
8:12 p.m.
On Mon, Jan 02, 2023 at 02:34:22PM -0500, Rich Salz wrote:
The difference is that "make" need only run
on your build systems.
Installation scripts need to run on customer systems.
Exactly. I let my team, against my judgement, use GNU make for our builds.
That was in house, the scripts are customer run.
--
---
Larry McVoy Retired to fishing http://www.mcvoy.com/lm/boat
8:24 p.m.
At 2023-01-02T14:34:22-0500, Rich Salz wrote:
The difference is that "make" need only run
on your build systems.
Installation scripts need to run on customer systems.
Surprisingly (not really), I don't agree with this, either.
make, like tsort, is a generally useful tool that people misconceive as
being limited in scope to traditional compile-and-link software
development.
A dab of imagination and familiarity with existing tooling can save one
much wheel reinvention.
That said I don't doubt that some vendors happily took these tools in
particular (along with the compiler, of course) and locked them up
within a "pro" or "development" tier.
But all that teaches us is something we should have already known: free
software is better than proprietary software. Why _shouldn't_ customer
systems have make(1)?
Regards,
Branden
8:41 p.m.
On Mon, Jan 02, 2023 at 02:24:22PM -0600, G. Branden Robinson wrote:
At 2023-01-02T14:34:22-0500, Rich Salz wrote:
If we wanted to use Makefiles in our customer run product, we would have
shipped it.
The difference is that "make" need only
run on your build systems.
Installation scripts need to run on customer systems.
Surprisingly (not really), I don't agree with this, either.
make, like tsort, is a generally useful tool that people misconceive as
being limited in scope to traditional compile-and-link software
development.
A dab of imagination and familiarity with existing tooling can save one
much wheel reinvention. 
9 p.m.
On Mon, Jan 2, 2023 at 3:41 PM Larry McVoy <lm(a)mcvoy.com> wrote:
On Mon, Jan 02, 2023 at 02:24:22PM -0600, G. Branden
Robinson wrote:
Couldn't you also ship a shell? That kinda seems like the simplest solution.
I used to be pretty dogmatic about adherence to 7th Ed tools and
syntax, but I think I'm like the frog in the pot where the temperature
is slowly rising: I suspect I've let things creep in that are not
portable to a universal intersection set, simply because I haven't
noticed, because I'm no longer trying to run things on very old
systems regularly. Will that script I wrote last week run on Ultrix?
Beats me; I haven't powered up the DECstation in 15 years. But
considering the issue raises a related question: practically speaking,
what do you define as a least common denominator? For example, is
`nawk` allowed, or just the traditional 7th Ed awk? How about extended
regular expressions? GNU coreutils `egrep` now spits out a warning
saying, "egrep is obsolescent; using grep -E" (I cannot adequately
express how irritating I find this). Does `tr` need the square
brackets? Etc. The shell by itself is just part of the equation, and
as a programming environment, /bin/sh is not all that interesting
unless complemented by a bunch of utilities, the portability of which
seems less guaranteed than it once was.
I actually think the shell is less of an issue; C is a bigger one for
me. About a year ago, I set up some emulated systems on my local ham
radio network (connected to AX.25 also!). Among them were Multics and
TOPS-20, but also a VAX running 4.3BSD and 7th Ed on a PDP-11. But
getting anything modern to build on either is a serious pain; speaking
of shells, I was able to find what is (I think) the last version of
tcsh to support 4.3 and get that to build.
Then again, having 7th Ed around forces me to think about shell script
portability again, so...there's that. :-)
- Dan C.
At 2023-01-02T14:34:22-0500, Rich Salz wrote:
If we wanted to use Makefiles in our customer run product, we would have
shipped it. The difference is that "make" need only
run on your build systems.
Installation scripts need to run on customer systems.
Surprisingly (not really), I don't agree with this, either.
make, like tsort, is a generally useful tool that people misconceive as
being limited in scope to traditional compile-and-link software
development.
A dab of imagination and familiarity with existing tooling can save one
much wheel reinvention.
9:19 p.m.
On Mon, Jan 2, 2023 at 4:07 PM Clem Cole <clemc(a)ccc.com> wrote:
On Mon, Jan 2, 2023 at 4:03 PM Dan Cross
<crossd(a)gmail.com> wrote:
Good question. But this sort of suggests that maybe it's not the shell
that's really the problem, but the entire milieu of all the tools that
we use when we write "shell scripts."
Couldn't you also ship a shell? That kinda
seems like the simplest solution.
Sure the problem is -- where do you stop? The result is the fork/mess called Gnu.
BUGS: Yes.
- Dan C.
10:54 p.m.
A shell is only as powerful as the constructs it can interact with. One can do logic all
day and night but if that logic isn't being put to work, it's just an exercise
in vanity. I think that's a good point that scripting problems may be a symptom of
the nature of the tools being used in them.
So the day job is pretty much an all Microsoft shop, from .NET code to IIS servers, not a
'$' prompt in sight usually. However, being a scientific operation, we have
quite a diverse crew of UNIX machines floating around out there resigned to their fates
sitting in the corner running the oldest equipment at their respective labs. Any time
I've had to interact with any of these I've been grateful for not taking various
post-Bourne shell quirks for granted. Not that these interactions are frequent, but I
find myself tapped often being "that UNIX guy" on the team. A couple examples
come to mind.
One facility is running two pieces of equipment the software for which runs on Digital
UNIX circa mid 90s. The data transfer mechanism was some FTP thing their previous
technical director had cooked up and one morning we had found it went down. The resident
IT guy there knew the basics of how to get in there, but hadn't really put the
process down anywhere. Luckily (security may think otherwise...) I was able to connect
over rsh with the password he provided and was able to modify the script for that exporter
to self-heal. I don't know what shell it was running beyond it spitting a
'$' out at me, but luckily didn't have to know.
The other is a lab using a data analysis system for, among others, HP-UX. I don't
recall the age of the version they were running on, but the system is called Target and
was eventually forked in a few directions when the original company closed up shop, our
group inheriting one of the forks. It's one of those things I hope I can archive
publicly someday, but not there yet. In any case, the technicians all had Windows PCs
with an X11 server setup for accessing the HP-UX machine, and it was running CDE if that
helps date it (unless that's still default). That system is in archive now as
we've replaced it with the current version of our fork which is now .NET. I was
surprised they didn't have more UNIX going on, but they had managed to get a lot of
their older equipment to work with some sort of Windows based solution one way or another.
Life finds a way...
I'm not sure where it stands as far as the sea of shells these days, but one
I've opted towards in system builds the past few years is dash. It's pretty
small and seems to hit all the marks as far as basic Bourne-ish behavior and POSIX stuff.
When I'm tinkering on a board and just need to toss a shell on top of a kernel, I go
for dash.
- Matt G.
------- Original Message -------
On Monday, January 2nd, 2023 at 1:19 PM, Dan Cross <crossd(a)gmail.com> wrote:
On Mon, Jan 2, 2023 at 4:07 PM Clem Cole clemc(a)ccc.com
wrote:
On Mon, Jan 2, 2023 at 4:03 PM Dan Cross
crossd(a)gmail.com wrote:
Good question. But this sort of suggests that maybe it's not the shell
that's really the problem, but the entire milieu of all the tools that
we use when we write "shell scripts."
Couldn't you also ship a shell? That kinda
seems like the simplest solution.
Sure the problem is -- where do you stop? The result is the fork/mess called Gnu.
BUGS: Yes.
- Dan C.
11:58 p.m.
segaloco via TUHS writes:
I think that's a good point that scripting
problems may be
a symptom of the nature of the tools being used in them.
I think that you're hinting at something different.
To the best of my recollection, scripting languages were originally
intended and used for the automation of repetitive personal tasks;
making it easier for users who found themselves typing the same
stuff over and over again. Somewhere along the line people forgot
how to use a compiler and began writing large systems in a variety
of roughly equivalent but incompatible interpreted languages. Can
one even boot linux without having several different incompatible
versions of Python installed today? So I don't think that it's the
nature of the tools; I think that it's people choosing the wrong
tools for the problems that they're trying to solve.
Jon
4 Jan
4 Jan
9 a.m.
On Jan 2, 2023, at 15:59, Jon Steinhart
<jon(a)fourwinds.com> wrote:
segaloco via TUHS writes:
I used to think this was a somewhat modern thing, at least evident by atrocities performed
by tcl & perl in the 1990s. But exposure to this group of weirdos has made me realize
that somewhat terrifying amounts of “infrastructure” code is built on little languages
like awk or ed scripts. And that any criticism of the horrible things I use Ruby for today
only really defends the place for awk and sed that those now hold and where they truly
excel.
But then I read https://www.bell-labs.com/usr/dmr/www/qed.html
I think that's a good point that scripting
problems may be
a symptom of the nature of the tools being used in them.
To the best of my recollection, scripting languages were originally
intended and used for the automation of repetitive personal tasks;
making it easier for users who found themselves typing the same
stuff over and over again. Somewhere along the line people forgot
how to use a compiler and began writing large systems in a variety
of roughly equivalent but incompatible interpreted languages. For a brief period in the 1970s, the GECOS QED served
us as a scripting language; it was in some ways analogous to the Awk or Perl of today. It
was used for such tasks as submitting batch jobs, formatting files for the printer,
collecting statistics on a file. A collection of macros to do various useful tasks was put
in a commonly available place.
I truly hope in my heart of hearts that ken had no inkling that someday QED would be used
as the scripting language to power http://www.qef.com/html/quickref.html. Not that QEF is
bad, just that any stones thrown at “you shouldn’t use $lang for that!” feel like they
must only strike via ricochet off QEF.
I can only imagine the shock and horror Larry McVoy would experience if one of his team
suggested replacing make with QEF.
2 Jan
2 Jan
10:43 p.m.
On Mon, 2 Jan 2023, Clem Cole wrote:
On Mon, Jan 2, 2023 at 4:03 PM Dan Cross
<crossd(a)gmail.com> wrote:
GNU is Embrace, Extend and Exterminate applied to Unix.
-uso.
Couldn't you also ship a shell? That kinda
seems like the simplest
solution.
Sure the problem is -- where do you stop?
The result is the fork/mess called Gnu.
9:08 p.m.
On Mon, Jan 2, 2023, at 13:00, Dan Cross wrote:
On Mon, Jan 2, 2023 at 3:41 PM Larry McVoy
<lm(a)mcvoy.com> wrote:
Then you have to ship documentation on how to properly install the shell. Also, add
guardrails to make sure that the correct shell is being loaded instead of someone getting
ambitious with their exec path.
cf every products trying to ship its own python, ruby, etc instead of accepting defeat and
shipping a statically build tool in a docker container in an OVA virtual machine image.
Obviously, that’s what should have been done with Bourne shell in the first place.
If we wanted to use Makefiles in our customer run product, we would have
shipped it.
Couldn't you also ship a shell? That kinda seems like the simplest solution.
9:15 p.m.
On Jan 2, 2023, at 2:08 PM, Joseph Holsten
<joseph(a)josephholsten.com> wrote:
On Mon, Jan 2, 2023, at 13:00, Dan Cross wrote:
You came very close to owing me a new keyboard there.
Adam
On Mon, Jan 2, 2023 at 3:41 PM Larry McVoy
<lm(a)mcvoy.com> wrote:
Then you have to ship documentation on how to properly install the shell. Also, add
guardrails to make sure that the correct shell is being loaded instead of someone getting
ambitious with their exec path.
cf every products trying to ship its own python, ruby, etc instead of accepting defeat
and shipping a statically build tool in a docker container in an OVA virtual machine
image.
Obviously, that’s what should have been done with Bourne shell in the first place.
If we wanted to use Makefiles in our customer run product, we would have
shipped it.
Couldn't you also ship a shell? That kinda seems like the simplest solution.
5:55 p.m.
On Jan 2, 2023, at 10:43 AM, Larry McVoy
<lm(a)mcvoy.com> wrote:
When you get to the commercial world, you'd be stunned to see how long
old machines last. If they are solving some problem, and they aren't
broken, nobody replaces them. I'll bet you anything there are still
SCO registers out there, I'll bet there are still PDP-11s out there.
If it ain't broke, don't fix it.
Well, sure, but this also comes down to at what point is losing some customers because
they refuse to upgrade to something from this millenium worth the cost you would incur to
be able to support their environment?
To be fair, stuff exactly like that was part of the bread-and-butter of Sine Nomine while
I was there (for all I know, still is), and I kinda feel like that's the right
answer: the places running old technology know they're doing so, and maybe
they're doing so for good reasons. But finding people willing and able to keep new
stuff working on those machines and OSes does get more expensive over time. So
there's an aftermarket in consultancies that are willing to take over maintenance of
things their vendors have abandoned. No company concerned with its long-term survival
*should* do that without, in parallel, pursuing an exit strategy towards something
actually-maintainable at not-exponentially-increasing cost, but of course the inability to
look beyond the next quarter is a fundamental flaw of financialization-driven capitalism.
And no one is saying you can't keep running your SCO cash registers forever on your
own recognizance, but when they break, you get to keep both pieces, and if they break in
such a way that suddenly your entire operation is being held hostage by some Russian
organized crime cartel, well, you should have priced that risk appropriately.
Adam
6:11 p.m.
Maybe this should go to COFF but Adam I fear you are falling into a tap
that is easy to fall into - old == unused
One of my favorite stores in the computer restoration community is from
5-10 years ago and the LCM+L in Seatle was restoring their CDC-6000 that
they got From Purdue. Core memory is difficult to get, so they made a
card and physical module that could plug into their system that is both
electrically and mechanically equivalent using modern semiconductors. A
few weeks later they announced that they had the system running and had
built this module. They got approached by the USAF asking if they could get
a copy of the design. Seems, there was still a least one CDC-6600 running
a particular critical application somewhere.
This is similar to the PDP-11s and Vaxen that are supposed to be still in
the hydroelectric grid [a few years ago the was an ad for an RSX and VMS
programmer to go to Canada running in the Boston newspapers - I know
someone that did a small amount of consulting on that one].
ᐧ
6:36 p.m.
On Mon, Jan 2, 2023 at 1:13 PM Clem Cole <clemc(a)ccc.com> wrote:
Maybe this should go to COFF but Adam I fear you are
falling into a tap that is easy to fall into - old == unused
One of my favorite stores in the computer restoration community is from 5-10 years ago
and the LCM+L in Seatle was restoring their CDC-6000 that they got From Purdue. Core
memory is difficult to get, so they made a card and physical module that could plug into
their system that is both electrically and mechanically equivalent using modern
semiconductors. A few weeks later they announced that they had the system running and
had built this module. They got approached by the USAF asking if they could get a copy of
the design. Seems, there was still a least one CDC-6600 running a particular critical
application somewhere.
This is similar to the PDP-11s and Vaxen that are supposed to be still in the
hydroelectric grid [a few years ago the was an ad for an RSX and VMS programmer to go to
Canada running in the Boston newspapers - I know someone that did a small amount of
consulting on that one].
One of my favorite stories along these lines is the train signalling
system in Melbourne, running on a "PDP-11". I quote PDP-11 because
that is now virtualized:
https://www.equicon.de/images/Virtualisierung/LegacyTrainControlSystemStabi…
Indeed older systems show up in surprising places. I was once on the
bridge of a US Naval vessel in the late '00s and saw a SPARCstaton 20
running Solaris (with the CDE login screen). I don't recall what it
was doing, but it was a tad surprising.
I do worry about legacy systems in critical situations, but then, I've
been in a firefight when the damned tactical computer with the satcomm
link rebooted and we didn't have VHF comms with the battlespace owner.
That was not particularly fun.
- Dan C.
6:48 p.m.
[Apologies for resending; I messed up and used the old Minnie address
for COFF in the Cc]
On Mon, Jan 2, 2023 at 1:36 PM Dan Cross <crossd(a)gmail.com> wrote:
On Mon, Jan 2, 2023 at 1:13 PM Clem Cole <clemc(a)ccc.com> wrote:
Maybe this should go to COFF but Adam I fear you
are falling into a tap that is easy to fall into - old == unused
One of my favorite stores in the computer restoration community is from 5-10 years ago
and the LCM+L in Seatle was restoring their CDC-6000 that they got From Purdue. Core
memory is difficult to get, so they made a card and physical module that could plug into
their system that is both electrically and mechanically equivalent using modern
semiconductors. A few weeks later they announced that they had the system running and
had built this module. They got approached by the USAF asking if they could get a copy of
the design. Seems, there was still a least one CDC-6600 running a particular critical
application somewhere.
This is similar to the PDP-11s and Vaxen that are supposed to be still in the
hydroelectric grid [a few years ago the was an ad for an RSX and VMS programmer to go to
Canada running in the Boston newspapers - I know someone that did a small amount of
consulting on that one].
One of my favorite stories along these lines is the train signalling
system in Melbourne, running on a "PDP-11". I quote PDP-11 because
that is now virtualized:
https://www.equicon.de/images/Virtualisierung/LegacyTrainControlSystemStabi…
Indeed older systems show up in surprising places. I was once on the
bridge of a US Naval vessel in the late '00s and saw a SPARCstaton 20
running Solaris (with the CDE login screen). I don't recall what it
was doing, but it was a tad surprising.
I do worry about legacy systems in critical situations, but then, I've
been in a firefight when the damned tactical computer with the satcomm
link rebooted and we didn't have VHF comms with the battlespace owner.
That was not particularly fun.
- Dan C.
6:18 p.m.
On Mon, Jan 02, 2023 at 10:55:51AM -0700, Adam Thornton wrote:
And no one is saying you can't keep running your
SCO cash registers
forever on your own recognizance, but when they break, you get to keep
both pieces, and if they break in such a way that suddenly your entire
operation is being held hostage by some Russian organized crime cartel,
well, you should have priced that risk appropriately.
Not my problem, I'm not running SCO cash registers, but the customer
might be. If they have enough money to interest me, then my problem is
having a product that works on their platform.
You argue with me all you want, you aren't going to win. Writing stuff
in a portable way is not that hard and once you have that figured out,
you just do it. It's the path to more revenue.
And I've fought this battle with some extremely intelligent and well
meaning engineers, absolutely in the top 1% of software engineers,
and they all caved. Sometimes it took me letting them off the leash
long enough to make something break, but they all caved eventually.
It's just more pleasant knowing that your stuff will work no matter
what /bin/sh is there. All the bash-ism in the world are less
valuable than "it just works". Don't get me wrong, I use bash-ism
for my personal use but I wouldn't let them get anywhere near a
product I needed to ship.
All that said, I'm done with this topic, feel free to have the last
word, I think I'm boring the smart people at this point.
4 Jan
4 Jan
3:20 a.m.
On Mon, Jan 2, 2023 at 12:43 PM Larry McVoy <lm(a)mcvoy.com> wrote:
I was supporting a commercial product in the early
2000's and there
were all sorts of systems then that had old shells. Yes, you could make
everything a 2 level thing where the first level finds the correct shell,
but that's just fuss. Just make things portable, it's not that hard
and it works everywhere.
Making shell scripts portable means not using pipelines, because given "foo
| bar", kshNN and zsh execute foo in a subshell and bar in the top-level
shell, whereas in other shells, both foo and bar execute in subshells.
(For this reason, Posix allows either behavior.) Not having pipelines is a
pretty drastic limitation.
3:31 a.m.
On Tue, Jan 3, 2023 at 10:22 PM John Cowan <cowan(a)ccil.org> wrote:
Making shell scripts portable means not using
pipelines, because given "foo | bar", kshNN and zsh execute foo in a subshell
and bar in the top-level shell, whereas in other shells, both foo and bar execute in
subshells. (For this reason, Posix allows either behavior.) Not having pipelines is a
pretty drastic limitation.
This came up at work just the other day:
echo ' hi ' | sed 's/^ *//;s/ *$//' | read bummer
echo $bummer
The behavior varies between ksh, zsh, bash, sh. Yay.
- Dan C.
4:16 a.m.
On Jan 3, 2023, at 7:31 PM, Dan Cross <crossd(a)gmail.com> wrote:
On Tue, Jan 3, 2023 at 10:22 PM John Cowan <cowan(a)ccil.org> wrote:
On this example sh, zsh, bash behave the same way on
freebsd. Rather than ban |, I think the lesson is to
*avoid* builtin commands in a pipeline that can affect
shell's environment. I only write /bin/sh scripts in
any case.
It is really unfortunate that infix | is so beguiling!
Except for that a Lisp or Scheme based shell would've
been fine and dandy! :-)/2
Making shell scripts portable means not using
pipelines, because given "foo | bar", kshNN and zsh execute foo in a subshell
and bar in the top-level shell, whereas in other shells, both foo and bar execute in
subshells. (For this reason, Posix allows either behavior.) Not having pipelines is a
pretty drastic limitation.
This came up at work just the other day:
echo ' hi ' | sed 's/^ *//;s/ *$//' | read bummer
echo $bummer
The behavior varies between ksh, zsh, bash, sh. Yay.
4:15 p.m.
On Tue, Jan 3, 2023 at 11:16 PM Bakul Shah <bakul(a)iitbombay.org> wrote:
On Jan 3, 2023, at 7:31 PM, Dan Cross
<crossd(a)gmail.com> wrote:
Interesting. In my testing, `bash` was the odd man out:
: doctor; zsh
: doctor; echo ' hi ' | sed 's/^ *//;s/ *$//' | read bummer
echo $bummer
hi
: doctor;
: doctor; bash
: doctor; echo ' hi ' | sed 's/^ *//;s/ *$//' | read bummer
echo $bummer
: doctor;
exit
: doctor; ksh
: doctor; echo ' hi ' | sed 's/^ *//;s/ *$//' | read bummer
: doctor; echo $bummer
hi
: doctor;
: doctor; sh
: doctor; echo ' hi ' | sed 's/^ *//;s/ *$//' | read bummer
: doctor; echo $bummer
hi
: doctor;
: doctor;
That's on illumos.
On OpenBSD, neither bash, ksh, or sh produce output, while zsh does. I
suspect ksh is pdksh and sh is a link to ksh there, though. ksh93
gives "hi".
On FreeBSD, neither bash nor sh produce "hi", but zsh and ksh93 do.
Same on DragonFly, macOS and Arch Linux. On 7th edition, I only tried
sh; empty output. 4.3BSD gives empty output for /bin/sh. For kicks,
neither tcsh nor csh understand `read`; I don't think that *csh has a
read-like builtin.
This is, of course, a silly example. Most script authors would do
something like:
bummer=`echo ' hi ' | sed 's/^ *//;s/ *$//'`
Or, perhaps even:
bummer=$(echo ' hi ' | sed 's/^ *//;s/ *$//')
Life's too short and at this point I prefer the latter, but more often
than not my fingers are wired for the former.
On Tue, Jan 3, 2023 at 10:22 PM John Cowan
<cowan(a)ccil.org> wrote:
On this example sh, zsh, bash behave the same way on
freebsd. Rather than ban |, I think the lesson is to
*avoid* builtin commands in a pipeline that can affect
shell's environment. I only write /bin/sh scripts in
any case. Making shell scripts portable means not using
pipelines, because given "foo | bar", kshNN and zsh execute foo in a subshell
and bar in the top-level shell, whereas in other shells, both foo and bar execute in
subshells. (For this reason, Posix allows either behavior.) Not having pipelines is a
pretty drastic limitation.
This came up at work just the other day:
echo ' hi ' | sed 's/^ *//;s/ *$//' | read bummer
echo $bummer
The behavior varies between ksh, zsh, bash, sh. Yay. It is really unfortunate that infix | is so
beguiling!
Except for that a Lisp or Scheme based shell would've
been fine and dandy! :-)/2
Whatever happened to `es` ? I guess it died on the vine. It seemed
like a nifty idea.
I made a concerted effort to use `rc` as my shell on Unix for a little
while. Without all the supporting plan9 machinery, though, I gave up.
Minimalism is great in an environment that can augment it with useful
functionality; outside of that environment, it can be kind of a pain
for work-a-day use.
- Dan C.
6:28 p.m.
What I see so far from this discussion: Ted has written a lot of useful
scripts, but they are explicitly for one shell (bash) on one kernel
(linux). I.e. portability is not a concern.
People immediately pointed out how easy it is to write non-portable shell
scripts, and others commented that it does not matter (to them).
Kernel portability matters to me. So does code that works without regard to
the shell someone is using. So does robust error handling. So does math.
These are all things that, in my experience, rule against complex shell
scripts.
I still prefer Go as my scripting language, but we all have our
preferences. And, yes, Go gets wordy, but it's a price I'm happy to pay for
good error handling. I much prefer an error message to 100 lines of python
backtrace.
On Wed, Jan 4, 2023 at 8:17 AM Dan Cross <crossd(a)gmail.com> wrote:
On Tue, Jan 3, 2023 at 11:16 PM Bakul Shah
<bakul(a)iitbombay.org> wrote:
Interesting. In my testing, `bash` was the odd man out:
: doctor; zsh
: doctor; echo ' hi ' | sed 's/^ *//;s/ *$//' | read bummer
echo $bummer
hi
: doctor;
: doctor; bash
: doctor; echo ' hi ' | sed 's/^ *//;s/ *$//' | read bummer
echo $bummer
: doctor;
exit
: doctor; ksh
: doctor; echo ' hi ' | sed 's/^ *//;s/ *$//' | read bummer
: doctor; echo $bummer
hi
: doctor;
: doctor; sh
: doctor; echo ' hi ' | sed 's/^ *//;s/ *$//' | read bummer
: doctor; echo $bummer
hi
: doctor;
: doctor;
That's on illumos.
On OpenBSD, neither bash, ksh, or sh produce output, while zsh does. I
suspect ksh is pdksh and sh is a link to ksh there, though. ksh93
gives "hi".
On FreeBSD, neither bash nor sh produce "hi", but zsh and ksh93 do.
Same on DragonFly, macOS and Arch Linux. On 7th edition, I only tried
sh; empty output. 4.3BSD gives empty output for /bin/sh. For kicks,
neither tcsh nor csh understand `read`; I don't think that *csh has a
read-like builtin.
This is, of course, a silly example. Most script authors would do
something like:
bummer=`echo ' hi ' | sed 's/^ *//;s/ *$//'`
Or, perhaps even:
bummer=$(echo ' hi ' | sed 's/^ *//;s/ *$//')
Life's too short and at this point I prefer the latter, but more often
than not my fingers are wired for the former.
On Jan 3, 2023, at 7:31 PM, Dan Cross
<crossd(a)gmail.com> wrote:
> On Tue, Jan 3, 2023 at 10:22 PM John Cowan <cowan(a)ccil.org> wrote:
>> Making shell scripts portable means not using pipelines, because
given
"foo | bar", kshNN and zsh execute foo in a subshell and bar in the
top-level shell, whereas in other shells, both foo and bar execute in
subshells. (For this reason, Posix allows either behavior.) Not having
pipelines is a pretty drastic limitation.
This came up at work just the other day:
echo ' hi ' | sed 's/^ *//;s/ *$//' | read bummer
echo $bummer
The behavior varies between ksh, zsh, bash, sh. Yay.
On this example sh, zsh, bash behave the same way on
freebsd. Rather than ban |, I think the lesson is to
*avoid* builtin commands in a pipeline that can affect
shell's environment. I only write /bin/sh scripts in
any case. It is really unfortunate that infix | is so
beguiling!
Except for that a Lisp or Scheme based shell would've
been fine and dandy! :-)/2
Whatever happened to `es` ? I guess it died on the vine. It seemed
like a nifty idea.
I made a concerted effort to use `rc` as my shell on Unix for a little
while. Without all the supporting plan9 machinery, though, I gave up.
Minimalism is great in an environment that can augment it with useful
functionality; outside of that environment, it can be kind of a pain
for work-a-day use.
- Dan C.
7:33 p.m.
On 1/4/23 11:15 AM, Dan Cross wrote:
On Tue, Jan 3, 2023 at 11:16 PM Bakul Shah
<bakul(a)iitbombay.org> wrote:
On Illumos, /bin/sh and /usr/xpg4/bin/sh are both links to ksh93.
ksh88/ksh93 and zsh are the only shells that unconditionally run the last
element of a pipeline in the current shell. Bash will do it if job control
is not enabled and the `lastpipe' option is enabled.
It's usually not a problem, but you do have to be aware of potential side
effects.
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU chet(a)case.edu http://tiswww.cwru.edu/~chet/
On Jan 3, 2023, at 7:31 PM, Dan Cross
<crossd(a)gmail.com> wrote:
Interesting. In my testing, `bash` was the odd man out:
: doctor; zsh
: doctor; echo ' hi ' | sed 's/^ *//;s/ *$//' | read bummer
echo $bummer
hi
: doctor;
: doctor; bash
: doctor; echo ' hi ' | sed 's/^ *//;s/ *$//' | read bummer
echo $bummer
: doctor;
exit
: doctor; ksh
: doctor; echo ' hi ' | sed 's/^ *//;s/ *$//' | read bummer
: doctor; echo $bummer
hi
: doctor;
: doctor; sh
: doctor; echo ' hi ' | sed 's/^ *//;s/ *$//' | read bummer
: doctor; echo $bummer
hi
: doctor;
: doctor;
That's on illumos. On Tue, Jan 3, 2023 at 10:22 PM John Cowan
<cowan(a)ccil.org> wrote:
On this example sh, zsh, bash behave the same way on
freebsd. Rather than ban |, I think the lesson is to
*avoid* builtin commands in a pipeline that can affect
shell's environment. I only write /bin/sh scripts in
any case. Making shell scripts portable means not using
pipelines, because given "foo | bar", kshNN and zsh execute foo in a subshell
and bar in the top-level shell, whereas in other shells, both foo and bar execute in
subshells. (For this reason, Posix allows either behavior.) Not having pipelines is a
pretty drastic limitation.
This came up at work just the other day:
echo ' hi ' | sed 's/^ *//;s/ *$//' | read bummer
echo $bummer
The behavior varies between ksh, zsh, bash, sh. Yay.
3:21 p.m.
Hi John,
Making shell scripts portable means not using
pipelines
No it doesn't...
because given "foo | bar", kshNN and zsh
execute foo in a subshell and
bar in the top-level shell, whereas in other shells, both foo and bar
execute in subshells.
...because a lot of the time that doesn't matter.
--
Cheers, Ralph.
3:54 p.m.
Indeed. MiniUnix had no hardware pipes, but the shell “simulated” it
by writing to a intermediate temp file.
------ Original Message ------
From "Ralph Corderoy" <ralph(a)inputplus.co.uk>
To tuhs(a)tuhs.org
Date 1/4/2023 10:21:02 AM
Subject [TUHS] Re: A few comments on porting the Bourne shell
Hi John,
Making shell scripts portable means not using
pipelines
No it doesn't...
because given "foo | bar", kshNN and
zsh execute foo in a subshell and
bar in the top-level shell, whereas in other shells, both foo and bar
execute in subshells.
...because a lot of the time that doesn't matter.
--
Cheers, Ralph.
2 Jan
2 Jan
5:55 p.m.
On Mon, Jan 2, 2023 at 11:51 AM Larry McVoy <lm(a)mcvoy.com> wrote:
I think it is less of an issue today but if I were
still supporting a multi
platform product, I'd still insist on it.
Yep, and it's so easy to do. It's just a discipline. @Adam I helped
create POSIX, so I get it.
But Larry nailed it. It is just cheaper and easier to be disciplined and
stick with Bourne for your scripts. *Teach your people the skills and you
save time and money in the long run.* It's that simple. As Larry says,
you never know and the problem is - when it happens, it tends to happen on
a short leash. If you have been disciplined, it's a non-problem. It's
really not that hard to use the V7 syntax. Everything you want/need to do
is there.
BTW: At Intel, a couple of years back (less than 3-5 years ago) we had a
site where we needed things to work on a specific target that was, shall we
say 'a bit custom' - V7 syntax was just fine for the installer - boy folks
were happy a few of us had been on their case to get rid of the bashism the
Millenials had tried to add (I'm not really sure POSIX.2 would have been
good enough -- maybe - but Bourne was fine].
FWIW: In my start-up times, under the same rules of being disciplined, as
VP of Engineering, I insisted, all C and C++ code was required to
'flex-e-lint' warning clean. I gave my folks a 3-week week slip to clean
everything up. I was cursed during that time. But guess what, the
outstanding bug list dropped to ⅒ of what it had been. Created quite a few
true believers. And we made those 3 weeks back before we were done.
Clem
ᐧ
3 Jan
3 Jan
5:08 p.m.
On 1/2/23, Clem Cole <clemc(a)ccc.com> wrote:
FWIW: In my start-up times, under the same rules of being disciplined, as
VP of Engineering, I insisted, all C and C++ code was required to
'flex-e-lint' warning clean. I gave my folks a 3-week week slip to clean
everything up. I was cursed during that time. But guess what, the
outstanding bug list dropped to ⅒ of what it had been. Created quite a few
true believers. And we made those 3 weeks back before we were done.
This was also the policy in DEC's compiler and software development
tools groups. This was mainly VMS stuff and we didn't have flex and
lint, but as Clem can attest the C and C++ compilers had very
extensive warning capabilities. It was group policy that all code had
to compile cleanly, without triggering diagnostic messages, before
check-in was allowed. Once you get through the initial cleanup of
existing code, this policy pays back big time in avoidance of nasty
Heisenbugs.
-Paul W.
7:19 p.m.
On Tue, Jan 3, 2023, 10:09 AM Paul Winalski <paul.winalski(a)gmail.com> wrote:
On 1/2/23, Clem Cole <clemc(a)ccc.com> wrote:
Not all fixes to appease warnings fix anything. But enough do that it's
worth it.
The one caveat here is that people must understand the warning and that any
change makes things better. There is nothing worse than just tossing a cast
in to brute force it, only to later discover it's the wrong cast or you
needed a different semantic change.
Warner
FWIW: In my start-up times, under the same rules of being disciplined, as
VP of Engineering, I insisted, all C and C++ code was required to
'flex-e-lint' warning clean. I gave my folks a 3-week week slip to
clean
everything up. I was cursed during that time.
But guess what, the
outstanding bug list dropped to ⅒ of what it had been. Created quite a
few
true believers. And we made those 3 weeks back
before we were done.
This was also the policy in DEC's compiler and software development
tools groups. This was mainly VMS stuff and we didn't have flex and
lint, but as Clem can attest the C and C++ compilers had very
extensive warning capabilities. It was group policy that all code had
to compile cleanly, without triggering diagnostic messages, before
check-in was allowed. Once you get through the initial cleanup of
existing code, this policy pays back big time in avoidance of nasty
Heisenbugs.
7:56 p.m.
All warnings are worthy of my consideration. I will usually learn
something about my compiler, my program, and often both. The compiler's
messages are valuable feedback in my quality work-style loop.
On 01/03/2023 12:19 PM, Warner Losh wrote:
On Tue, Jan 3, 2023, 10:09 AM Paul Winalski <paul.winalski(a)gmail.com
<mailto:paul.winalski@gmail.com>> wrote:
On 1/2/23, Clem Cole <clemc(a)ccc.com <mailto:clemc@ccc.com>> wrote:
FWIW: In my start-up times, under the same rules of being
disciplined, as
VP of Engineering, I insisted, all C and C++ code
was required to
'flex-e-lint' warning clean. I gave my folks a 3-week week
slip to clean
everything up. I was cursed during that time.
But guess
what, the
outstanding bug list dropped to ⅒ of what it had
been. Created
quite a few
true believers. And we made those 3 weeks back
before we were done.
This was also the policy in DEC's compiler and software development
tools groups. This was mainly VMS stuff and we didn't have flex and
lint, but as Clem can attest the C and C++ compilers had very
extensive warning capabilities. It was group policy that all code had
to compile cleanly, without triggering diagnostic messages, before
check-in was allowed. Once you get through the initial cleanup of
existing code, this policy pays back big time in avoidance of nasty
Heisenbugs.
Not all fixes to appease warnings fix anything. But enough do that
it's worth it.
The one caveat here is that people must understand the warning and
that any change makes things better. There is nothing worse than just
tossing a cast in to brute force it, only to later discover it's the
wrong cast or you needed a different semantic change.
Warner
8:21 p.m.
On Tue, 3 Jan 2023, Warner Losh wrote:
Not all fixes to appease warnings fix anything. But
enough do that it's
worth it.
I consider warnings to be potential errors, and act accordingly; one place
where I worked insisted that code could not be checked in unless it passed
"-Wall" (no guarantee of course, but a good start).
The one caveat here is that people must understand the
warning and that
any change makes things better. There is nothing worse than just tossing
a cast in to brute force it, only to later discover it's the wrong cast
or you needed a different semantic change.
Yeah, that's just painting over the cracks; it's like replacing a fuse
that keeps blowing, or rebooting a box without fixing the bug (what I call
the M$ Solution).
-- Dave
9:47 p.m.
Bring this back to TUHS relevance...
On Tue, Jan 3, 2023 at 2:43 PM Warner Losh <imp(a)bsdimp.com> wrote:
The one caveat here is that people must understand the
warning and that
any change makes things better. There is nothing worse than just tossing a
cast in to brute force it, only to later discover it's the wrong cast or
you needed a different semantic change.
Most certainly -- It's why I >>loved<< Gimpel's flex-e-lint
product and
as Paul pointed out, Judy Ward's messages from the DEC Gem compiler - both
were the two best I ever ran into in giving you real information about what
was happening.
I also tell a story from my DEC time. After the GEM compiler was released
and before most other vendors were 64-bits, the ISVs were first starting to
do their ports to Alpha. My complaining ensued. We would discover from
numerous ISVs that after the Alpha port was complete, their bug count
dropped - why because the ISV's code has been kinda nasty and the older
compilers had been silent about it they had assumed the ILP32 model.
Alphas, using LP64, could not be. Judy would find things and say -- what a
minute -- you want me to do what with that and issued a fairly detailed
warning (which was the key - she explained what the issue was). Often the
32 to 64-bit nature forced the programmers at the ISV's to rethink how there
were actually declaring things to make the code clearer, simpler, better,
*etc*. The classic rewrite never happens unless you are forced too.
I remember going to a Supercomputer conference and talking with the
developers at one the ISVs who I will not name. He thanks me. He said his
team has been arguing with their management for years to redo the UNIX
support library. The Tru64 port was what finally allowed them to do it.
But it took 9 months which pissed off his boss. But when it was completed,
and pass all the tests on the Alpha, it just recompiled on Solaris, AIX,
and HP-UX - which had never happened before. He could not believe what a
great compiler we had.
I've always said the Alpha was the greatest gift to Sun and Intel in the
commercial SW world because it forced the ISV to clean up their act before
they ever saw those processors and Sun/Intel ports were piece of cake. It
was not that porting to Alpha was difficult -- it was cleaning up your own
mess.
Remember we had already gone through this in the PDP-11 ILP16 to Vax ILP32
transition but it is funny how history repeated itself.
ᐧ
9:51 p.m.
typo: s/My complaining/Much complaining/
sigh.. dyslex-r-me
ᐧ
On Tue, Jan 3, 2023 at 4:47 PM Clem Cole <clemc(a)ccc.com> wrote:
Bring this back to TUHS relevance...
On Tue, Jan 3, 2023 at 2:43 PM Warner Losh <imp(a)bsdimp.com> wrote:
The one caveat here is that people must
understand the warning and that
any change makes things better. There is nothing worse than just tossing a
cast in to brute force it, only to later discover it's the wrong cast or
you needed a different semantic change.
Most certainly -- It's why I >>loved<< Gimpel's
flex-e-lint product and
as Paul pointed out, Judy Ward's messages from the DEC Gem compiler - both
were the two best I ever ran into in giving you real information about what
was happening.
I also tell a story from my DEC time. After the GEM compiler was
released and before most other vendors were 64-bits, the ISVs were first
starting to do their ports to Alpha. My complaining ensued. We would discover
from numerous ISVs that after the Alpha port was complete, their bug count
dropped - why because the ISV's code has been kinda nasty and the older
compilers had been silent about it they had assumed the ILP32 model.
Alphas, using LP64, could not be. Judy would find things and say -- what
a minute -- you want me to do what with that and issued a fairly detailed
warning (which was the key - she explained what the issue was). Often
the 32 to 64-bit nature forced the programmers at the ISV's to rethink
how there were actually declaring things to make the code clearer,
simpler, better, *etc*. The classic rewrite never happens unless you
are forced too.
I remember going to a Supercomputer conference and talking with the
developers at one the ISVs who I will not name. He thanks me. He said his
team has been arguing with their management for years to redo the UNIX
support library. The Tru64 port was what finally allowed them to do it.
But it took 9 months which pissed off his boss. But when it was completed,
and pass all the tests on the Alpha, it just recompiled on Solaris, AIX,
and HP-UX - which had never happened before. He could not believe what a
great compiler we had.
I've always said the Alpha was the greatest gift to Sun and Intel in the
commercial SW world because it forced the ISV to clean up their act before
they ever saw those processors and Sun/Intel ports were piece of cake. It
was not that porting to Alpha was difficult -- it was cleaning up your own
mess.
Remember we had already gone through this in the PDP-11 ILP16 to Vax
ILP32 transition but it is funny how history repeated itself.
ᐧ
31 Dec
31 Dec
4:41 a.m.
On Friday, 30 December 2022 at 19:59:31 -0800, Larry McVoy wrote:
Over and over, I won the "straight bourne shell
only" battle. So I agree,
if you want /bin/sh to work, Bourne shell for the win.
Agreed.
For a login shell, bash is my shell of choice.
It's bloated but I'm
typing this on a 5 year old Lenova X1 Carbon with 16GB of memory and
4 cores and it's fine. It was fine a 133mhz Pentium.
I've been using bash since 16 MHz i386s. People told me that it was
bloated, but somehow I didn't notice.
Somehow this reminds me of the expansion of EMACS: Eight Megabytes And
Continually Swapping. 8 MB? Nowadays? It shows how old these
prejudices are. Now my emacs processes have round 30 MB of resident
memory and are dwarfed by 2 GB web browsers. bash comes in at a
little over 2 MB, just not worth talking about.
Greg
--
Sent from my desktop computer.
Finger grog(a)lemis.com for PGP public key.
See complete headers for address and phone numbers.
This message is digitally signed. If your Microsoft mail program
reports problems, please read http://lemis.com/broken-MUA.php
30 Dec
30 Dec
8:20 p.m.
Chet Ramey on 30.12.2022 20:51:
Arnold Robbins built on that work and ported the
v8-v10 shells to modern
Linux versions. (I am sorry, I do not have a link right now.)
And btw, 8th ed (http://man.cat-v.org/unix_8th/1/sh) even added a simple
history mechanism with the "=(1)" command
(https://www.in-ulm.de/~mascheck/bourne/v8/=.html).
Sven
8:49 p.m.
I was just happy when VR2 came out and found someone had undone all
those macros that were in the original code.
I hacked Berkeley Job Control as well as command line editing into it
(KSH hadn’t seen the light of day outside the labs at that point).
I subsequently had a nice talk with Korn at a USENIX. I also sat down
with a couple of the guys trying to implement their own shell
independent of the ATT code and explained to them how Berkeley job
control works. It’s for that reason my name shows up in a lot of the
early Linux docs.
Amusingly, I’d forgotten all about this stuff until one day I was
sitting at a MIPS workstation (MIPS branded, not the DEC spim).
Without thinking about it, I typed “fg” at the shell prompt.
“Job control not Enabled,” it said. Hey! That sounds like one of my
messages. “set -J” I type. “Job control enabled.”
Hey! This is a “Ron shell” as it was known at BRL. Turns out it went
out on the Mach distros so I’ve found it on all kinds of things like the
NeXT etc…
------ Original Message ------
From "Sven Mascheck" <mascheck(a)in-ulm.de>
To tuhs(a)tuhs.org
Date 12/30/2022 3:20:28 PM
Subject [TUHS] Re: A few comments on porting the Bourne shell
Chet Ramey on 30.12.2022 20:51:
Arnold Robbins built on that work and ported the
v8-v10 shells to modern
Linux versions. (I am sorry, I do not have a link right now.)
And btw, 8th ed (http://man.cat-v.org/unix_8th/1/sh) even added a simple history mechanism
with the "=(1)" command (https://www.in-ulm.de/~mascheck/bourne/v8/=.html).
Sven
8:52 p.m.
That "someone" was Steve Bourne. At least, the version he handed me when I
built up the v8 shell was from him and clean of ELIFs.
-rob
On Sat, Dec 31, 2022 at 7:50 AM Ron Natalie <ron(a)ronnatalie.com> wrote:
I was just happy when VR2 came out and found someone
had undone all
those macros that were in the original code.
I hacked Berkeley Job Control as well as command line editing into it
(KSH hadn’t seen the light of day outside the labs at that point).
I subsequently had a nice talk with Korn at a USENIX. I also sat down
with a couple of the guys trying to implement their own shell
independent of the ATT code and explained to them how Berkeley job
control works. It’s for that reason my name shows up in a lot of the
early Linux docs.
Amusingly, I’d forgotten all about this stuff until one day I was
sitting at a MIPS workstation (MIPS branded, not the DEC spim).
Without thinking about it, I typed “fg” at the shell prompt.
“Job control not Enabled,” it said. Hey! That sounds like one of my
messages. “set -J” I type. “Job control enabled.”
Hey! This is a “Ron shell” as it was known at BRL. Turns out it went
out on the Mach distros so I’ve found it on all kinds of things like the
NeXT etc…
------ Original Message ------
From "Sven Mascheck" <mascheck(a)in-ulm.de>
To tuhs(a)tuhs.org
Date 12/30/2022 3:20:28 PM
Subject [TUHS] Re: A few comments on porting the Bourne shell
Chet Ramey on 30.12.2022 20:51:
history
mechanism with the "=(1)" command (
https://www.in-ulm.de/~mascheck/bourne/v8/=.html).
Arnold Robbins built on that work and ported the
v8-v10 shells to modern
Linux versions. (I am sorry, I do not have a link right now.)
And btw, 8th ed (http://man.cat-v.org/unix_8th/1/sh) even added a simple
Sven
8:53 p.m.
Ron Natalie writes:
I was just happy when VR2 came out and found someone
had undone all
those macros that were in the original code.
On an amusing note, a while ago I had polled folks on a different
mailing list on items to include in a starter tool set for my daughter.
Steve:
Hi Jon, highly recommend DeWalt 20V drill, here is my checklist for my boat
tools. Might be something there to add. Some clearly not but didn't scrub.
Jon:
Thanks. I was thinking that I should also get her one of those adapters that
connects a { to a BEGIN :-)
Steve:
Looks like this
:-}
1 Jan
1 Jan
10:44 a.m.
Chet Ramey <chet.ramey(a)case.edu> wrote:
On 12/30/22 1:25 PM, Paul Ruizendaal wrote:
Sorry to say, it wasn't me. Geoff Collyer made the v9 sh portable
and it's available from his web site.
I did do stuff with the shell back in the 80s:
- I back-ported Ron Natalie's job control into the 4.2 BSD sh,
along with a history mechanism I wrote from scratch that was
inspired by csh. I posted the diffs in comp.sources.unix.
- A few very minor features were added to ksh at my suggestion.
ISTR that one was `set -A array-name value ...' but I don't see that
in current ksh93 doc.
- I was a reviewer for both editions of David Korn's book on ksh.
- I contributed to pdksh, although I don't remember what. The pdksh
doc lists my name though.
- In the early 90s I did a little banging on the vi mode in the
readline librar so that bash's vi mode would be closer to what
I'd been used to in ksh.
In the late 80s I got interested in awk and moved almost all my
personal scripting to awk, although I still do shell work at $DAYJOB
or for personal use as needed.
Arnold
London and Reiser report about porting the shell that “it required by
far the largest conversion effort of any supposedly portable program,
for the simple reason that it is not portable.” By the time of SysIII
this is greatly improved, but also in porting the SysIII user land it
was the most complex of the set so far.
Have you read
http://www.collyer.net/who/geoff/sh.tour.pdf
and looked at http://www.collyer.net/who/geoff/v7sh.tar ?
In the limited literature on Bourne Shell porting, this is authoritative.
Arnold Robbins built on that work and ported the v8-v10 shells to modern
Linux versions. (I am sorry, I do not have a link right now.)
11:28 a.m.
I wrote:
I did do stuff with the shell back in the 80s:
I also recall that there was some USENET discussion in which
Rob Pike described how the 8th edition shell changed the search
order to have functions come before non-special built-ins, with
the addition of "builtin" and "whatis" commands.
Rob was kind enough to send me the sh.1 man page from the 8th edition
(which I still have somewhere), and I made this change in the S5r2
shell. I don't know if I posted diffs or not.
In any case, as part of implementing whatis, I wrote a pretty printer
for shell functions. (Getting 'elif' right needed some extra code.) I
offered it to Rob, who politely declined, indicating that he had his own.
Ah, those were the days... :-)
Arnold
3 Jan
3 Jan
4:34 p.m.
On 1/1/23 6:28 AM, arnold(a)skeeve.com wrote:
I also recall that there was some USENET discussion in
which
Rob Pike described how the 8th edition shell changed the search
order to have functions come before non-special built-ins, with
the addition of "builtin" and "whatis" commands.
I have this somewhere. It's a rocking back-and-forth between Rob and
David Korn about shell philosophy. Well worth the read if you can find it.
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU chet(a)case.edu http://tiswww.cwru.edu/~chet/
3:06 p.m.
On 1/1/23 5:44 AM, arnold(a)skeeve.com wrote:
Chet Ramey <chet.ramey(a)case.edu> wrote:
My bad, credit to Geoff.
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU chet(a)case.edu http://tiswww.cwru.edu/~chet/
On 12/30/22 1:25 PM, Paul Ruizendaal wrote:
Sorry to say, it wasn't me. Geoff Collyer made the v9 sh portable
and it's available from his web site.
London and Reiser report about porting the shell that “it required by
far the largest conversion effort of any supposedly portable program,
for the simple reason that it is not portable.” By the time of SysIII
this is greatly improved, but also in porting the SysIII user land it
was the most complex of the set so far.
Have you read
http://www.collyer.net/who/geoff/sh.tour.pdf
and looked at http://www.collyer.net/who/geoff/v7sh.tar ?
In the limited literature on Bourne Shell porting, this is authoritative.
Arnold Robbins built on that work and ported the v8-v10 shells to modern
Linux versions. (I am sorry, I do not have a link right now.)
30 Dec
30 Dec
7:57 p.m.
I'll have to double check later but I'm fairly certain the remaining L/R cheats
are gone by SysV. From what I can tell much of that portability work may have been done
prior to the V7 release code base we're familiar with, as I did some comparison and
found only one significant change between V7 and 32V code as I know it at least. Either
the claims of portability issues came between 32V and System III (meaning the shell was
accepted as "broken"? in 32V) or the code we actually see in V7 has already been
tidied up significantly and doesn't represent the "non-portable" version
lamented in the famous quote. Does this observation hold with reality? Is there an
earlier, more PDP-11 bound version of the Bourne Shell out there? I seem to recall
reading something about some bits of it even being in assembly at one point, but
can't remember the quote source.
- Matt G.
------- Original Message -------
On Friday, December 30th, 2022 at 10:25 AM, Paul Ruizendaal <pnr(a)planet.nl> wrote:
London and Reiser report about porting the shell that
“it required by far the largest conversion effort of any supposedly portable program, for
the simple reason that it is not portable.” By the time of SysIII this is greatly
improved, but also in porting the SysIII user land it was the most complex of the set so
far.
There were three aspects that I found noteworthy:
1. London/Reiser apparently felt strongly about a property of casts. The code argues that
casting an l-value should not convert it into a r-value:
<quote from "mode.h">
/* the following nonsense is required
* because casts turn an Lvalue
* into an Rvalue so two cheats
* are necessary, one for each context.
*/
union { int _cheat;};
#define Lcheat(a) ((a)._cheat)
#define Rcheat(a) ((int)(a))
<endquote>
However, Lcheat is only used in two places (in service.c), to set and to clear a flag in
a pointer. Interestingly, the 32V code already replaces one of these instances with a
regular r-value cast. So far, I’d never thought about this aspect of casts. I stumbled
across it, because the Plan 9 compiler did not accept the Lcheat expansion as valid C.
2. On the history of dup2
The shell code includes the following:
<quote from “io.c”>
rename(f1,f2)
REG INT f1, f2;
{
#ifdef RES /* research has different sys calls from TS */
IF f1!=f2
THEN dup(f1|DUPFLG, f2);
close(f1);
IF f2==0 THEN ioset|=1 FI
FI
#else
INT fs;
IF f1!=f2
THEN fs = fcntl(f2,1,0);
close(f2);
fcntl(f1,0,f2);
close(f1);
IF fs==1 THEN fcntl(f2,2,1) FI
IF f2==0 THEN ioset|=1 FI
FI
#endif
}
<endquote>
I’ve check the 8th edition source, and indeed it supports using DUPFLG to signal to dup()
that it really is dup2(). I had earlier wondered why dup2() did not appear in research
until 10th edition, but now that is clear. It would seem that the dup of 8th edition is a
direct ancestor to dup() in Plan 9. I wonder why this way of doing things never caught on
in the other Unices.
3. Halfway to demand paging
I stumbled across this one because I had a bug in my signal handling. From early days
onwards, Unix supported dynamically growing the stack allocation, which arguably is a
first step towards building the mechanisms for demand paging. It appears that the Bourne
shell made another step, catching page faults and expanding the data/bss allocation
dynamically:
<quote from “fault.c”>
VOID fault(sig)
REG INT sig;
{
signal(sig, fault);
IF sig==MEMF
THEN IF setbrk(brkincr) == -1
THEN error(nospace);
FI
ELIF ...
<endquote>
This was already present in 7th edition, so it is by no means new in 32V or SysIII -- it
had just escaped my attention as a conceptual step in the development of Unix memory
handling.
31 Dec
31 Dec
12:55 p.m.
The "assembly code in the Bourne shell" comment is in the same London/Reiser
paper. The full quote is:
"The (Bourne) shell is the standard user command interpreter. It required by far the
largest conversion effort of any supposedly portable program, for the simple reason that
it is not portable. Critical portions are coded in assembly language and had to be
painstakingly rewritten. The shell uses its own sbrk which is functionally different from
the standard routine in libc. The shell wants the routine which fields a signal to be
passed a parameter giving the number of the signal being caught; signal was also a private
rou- tine. This was handled by having the operating system provide the parameter in the
first place, doing away with the private code for signal. The code in fixargs (for
constructing the argument list to an exec system call) had to be diddled."
The files in the V7 tree on the Tuhs website are dated January 1979, so it would seem that
the fixes for 32V were immediately taken back to Research. As you point out, this means
that the comments above do not refer to the well known source code, but to a predecessor
of that (which I don’t think survived).
Despite all the criticism voiced above, I think it is well understood that the original
Bourne shell is an amazing piece of work that managed to fit an enormous amount of
functionality into a cramped address space. Its longevity attests to that. That its
internals became difficult to understand is par for the course -- the 1980’s in essence
needed a Lions commentary on sh.
On 30 Dec 2022, at 20:57, segaloco
<segaloco(a)protonmail.com> wrote:
I'll have to double check later but I'm fairly certain the remaining L/R cheats
are gone by SysV. From what I can tell much of that portability work may have been done
prior to the V7 release code base we're familiar with, as I did some comparison and
found only one significant change between V7 and 32V code as I know it at least. Either
the claims of portability issues came between 32V and System III (meaning the shell was
accepted as "broken"? in 32V) or the code we actually see in V7 has already been
tidied up significantly and doesn't represent the "non-portable" version
lamented in the famous quote. Does this observation hold with reality? Is there an
earlier, more PDP-11 bound version of the Bourne Shell out there? I seem to recall
reading something about some bits of it even being in assembly at one point, but
can't remember the quote source.
- Matt G.
------- Original Message -------
On Friday, December 30th, 2022 at 10:25 AM, Paul Ruizendaal <pnr(a)planet.nl> wrote:
> London and Reiser report about porting the shell that “it required by far the largest
conversion effort of any supposedly portable program, for the simple reason that it is not
portable.” By the time of SysIII this is greatly improved, but also in porting the SysIII
user land it was the most complex of the set so far.
>
> There were three aspects that I found noteworthy:
>
> 1. London/Reiser apparently felt strongly about a property of casts. The code argues
that casting an l-value should not convert it into a r-value:
>
> <quote from "mode.h">
>
> /* the following nonsense is required
> * because casts turn an Lvalue
> * into an Rvalue so two cheats
> * are necessary, one for each context.
> */
> union { int _cheat;};
> #define Lcheat(a) ((a)._cheat)
> #define Rcheat(a) ((int)(a))
> <endquote>
>
>
> However, Lcheat is only used in two places (in service.c), to set and to clear a flag
in a pointer. Interestingly, the 32V code already replaces one of these instances with a
regular r-value cast. So far, I’d never thought about this aspect of casts. I stumbled
across it, because the Plan 9 compiler did not accept the Lcheat expansion as valid C.
>
> 2. On the history of dup2
>
> The shell code includes the following:
>
> <quote from “io.c”>
>
> rename(f1,f2)
> REG INT f1, f2;
> {
> #ifdef RES /* research has different sys calls from TS */
> IF f1!=f2
> THEN dup(f1|DUPFLG, f2);
> close(f1);
> IF f2==0 THEN ioset|=1 FI
> FI
> #else
> INT fs;
> IF f1!=f2
> THEN fs = fcntl(f2,1,0);
> close(f2);
> fcntl(f1,0,f2);
> close(f1);
> IF fs==1 THEN fcntl(f2,2,1) FI
> IF f2==0 THEN ioset|=1 FI
> FI
> #endif
> }
> <endquote>
>
>
> I’ve check the 8th edition source, and indeed it supports using DUPFLG to signal to
dup() that it really is dup2(). I had earlier wondered why dup2() did not appear in
research until 10th edition, but now that is clear. It would seem that the dup of 8th
edition is a direct ancestor to dup() in Plan 9. I wonder why this way of doing things
never caught on in the other Unices.
>
> 3. Halfway to demand paging
>
> I stumbled across this one because I had a bug in my signal handling. From early days
onwards, Unix supported dynamically growing the stack allocation, which arguably is a
first step towards building the mechanisms for demand paging. It appears that the Bourne
shell made another step, catching page faults and expanding the data/bss allocation
dynamically:
>
> <quote from “fault.c”>
>
> VOID fault(sig)
> REG INT sig;
> {
> signal(sig, fault);
> IF sig==MEMF
> THEN IF setbrk(brkincr) == -1
> THEN error(nospace);
> FI
> ELIF ...
> <endquote>
>
>
> This was already present in 7th edition, so it is by no means new in 32V or SysIII --
it had just escaped my attention as a conceptual step in the development of Unix memory
handling.
>
1 Jan
1 Jan
2:55 a.m.
On Sat, Dec 31, 2022 at 5:55 AM Paul Ruizendaal <pnr(a)planet.nl> wrote:
The "assembly code in the Bourne shell"
comment is in the same
London/Reiser paper. The full quote is:
"The (Bourne) shell is the standard user command interpreter. It required
by far the largest conversion effort of any supposedly portable program,
for the simple reason that it is not portable. Critical portions are coded
in assembly language and had to be painstakingly rewritten. The shell uses
its own sbrk which is functionally different from the standard routine in
libc. The shell wants the routine which fields a signal to be passed a
parameter giving the number of the signal being caught; signal was also a
private rou- tine. This was handled by having the operating system provide
the parameter in the first place, doing away with the private code for
signal. The code in fixargs (for constructing the argument list to an exec
system call) had to be diddled."
The files in the V7 tree on the Tuhs website are dated January 1979, so it
would seem that the fixes for 32V were immediately taken back to Research.
As you point out, this means that the comments above do not refer to the
well known source code, but to a predecessor of that (which I don’t think
survived).
We have ample evidence that V7 was really something more akin to a rolling
release. Let me explain: We know from the leaked '50 changes' tape that
many of the features were set earlier rather than later. This leaked in
1978 (if my notes are right), but I found references to it from as early as
November 1976 in
http://www.toad.com/early-usenix-newsletters/197611-unix-news-n11.pdf. This
was 18 months after V6 was released, but over 2 years before V7 was
released. In addition, we know from the AUUS newsletters in the archive
document that the V7 release process process took a while to get through
AT&T's legal department (IIRC a year, but I've not gone back to the AUUS
newsletters to refresh my recollection). A big push of V7 was to make it
portable as well (with AT&T doing an Interdata 8/32 port themselves, as
well as at least looking at the Wollongong Interdata 7/32 port and the
Harvard VM/370 port). In talking to Kirk and others that have been around
from approximately that time, 32V was widely viewed as V7 for Vaxen. We can
see evidence in the surviving 32V files of evolution from the 'PDP-11-like
swapping to a more sophisticated paging algorithm' since we have the
slowsys directory. It's my contention, as someone that coded in the era
before good source code control, that it's evidence that somebody got it
working, then renamed/copied it to slowsys while they got paging working so
they could build either kernel for A/B testing. Kirk has also told me that
the 32V port was started well in advance of V7's release to be both a
useful product inside of Bell Labs (since Vaxen were starting to appear) as
well as to prove that V7 was portable enough. I'll be the first to admit
this is at best conjecture that matches available facts, artifacts and old
timers recollections (sorry Kirk), but that we have no direct evidence for.
It also allowed the 3BSD efforts to get going before the official V7
release due to the close ties between Bell Labs and Berkeley and the DARPA
project around Unix.
I believe that we can conclude that the original 'hard to port' Bourne
shell was produced around the time of the 50 changes tape, give or take.
And that all the unix porting efforts that pre-dated the V7 release rolled
what appeared in 32V into V7 to reduce the amount of pdp-11 assembler. And
those efforts are what we read about in the paper.
It also goes a ways to explain the 32V meme of 'it was pdp11 swapping'
because originally, in a version we no longer have, it was. But many of the
32V tapes that we have represent a later version where that had been
abandoned in favor of what would evolve into System III's and later paging
code.
Despite all the criticism voiced above, I think it is
well understood that
the original Bourne shell is an amazing piece of work that managed to fit
an enormous amount of functionality into a cramped address space. Its
longevity attests to that. That its internals became difficult to
understand is par for the course -- the 1980’s in essence needed a Lions
commentary on sh.
Totally agree with that...
Warner
On 30 Dec 2022, at 20:57, segaloco
<segaloco(a)protonmail.com> wrote:
I'll have to double check later but I'm fairly certain the remaining L/R
cheats are gone by SysV. From what I can tell much of that portability
work may have been done prior to the V7 release code base we're familiar
with, as I did some comparison and found only one significant change
between V7 and 32V code as I know it at least. Either the claims of
portability issues came between 32V and System III (meaning the shell was
accepted as "broken"? in 32V) or the code we actually see in V7 has already
been tidied up significantly and doesn't represent the "non-portable"
version lamented in the famous quote. Does this observation hold with
reality? Is there an earlier, more PDP-11 bound version of the Bourne
Shell out there? I seem to recall reading something about some bits of it
even being in assembly at one point, but can't remember the quote source.
- Matt G.
------- Original Message -------
On Friday, December 30th, 2022 at 10:25 AM, Paul Ruizendaal <
pnr(a)planet.nl>
wrote:
> London and Reiser report about porting the shell that “it required by
far the
largest conversion effort of any supposedly portable program, for
the simple reason that it is not portable.” By the time of SysIII this is
greatly improved, but also in porting the SysIII user land it was the most
complex of the set so far.
>
> There were three aspects that I found noteworthy:
>
> 1. London/Reiser apparently felt strongly about a property of casts.
The code
argues that casting an l-value should not convert it into a
r-value:
>
> <quote from "mode.h">
>
> /* the following nonsense is required
> * because casts turn an Lvalue
> * into an Rvalue so two cheats
> * are necessary, one for each context.
> */
> union { int _cheat;};
> #define Lcheat(a) ((a)._cheat)
> #define Rcheat(a) ((int)(a))
> <endquote>
>
>
> However, Lcheat is only used in two places (in service.c), to set and
to clear
a flag in a pointer. Interestingly, the 32V code already replaces
one of these instances with a regular r-value cast. So far, I’d never
thought about this aspect of casts. I stumbled across it, because the Plan
9 compiler did not accept the Lcheat expansion as valid C.
>
> 2. On the history of dup2
>
> The shell code includes the following:
>
> <quote from “io.c”>
>
> rename(f1,f2)
> REG INT f1, f2;
> {
> #ifdef RES /* research has different sys calls from TS */
> IF f1!=f2
> THEN dup(f1|DUPFLG, f2);
> close(f1);
> IF f2==0 THEN ioset|=1 FI
> FI
> #else
> INT fs;
> IF f1!=f2
> THEN fs = fcntl(f2,1,0);
> close(f2);
> fcntl(f1,0,f2);
> close(f1);
> IF fs==1 THEN fcntl(f2,2,1) FI
> IF f2==0 THEN ioset|=1 FI
> FI
> #endif
> }
> <endquote>
>
>
> I’ve check the 8th edition source, and indeed it supports using DUPFLG
to
signal to dup() that it really is dup2(). I had earlier wondered why
dup2() did not appear in research until 10th edition, but now that is
clear. It would seem that the dup of 8th edition is a direct ancestor to
dup() in Plan 9. I wonder why this way of doing things never caught on in
the other Unices.
>
> 3. Halfway to demand paging
>
> I stumbled across this one because I had a bug in my signal handling.
From
early days onwards, Unix supported dynamically growing the stack
allocation, which arguably is a first step towards building the mechanisms
for demand paging. It appears that the Bourne shell made another step,
catching page faults and expanding the data/bss allocation dynamically:
>
> <quote from “fault.c”>
>
> VOID fault(sig)
> REG INT sig;
> {
> signal(sig, fault);
> IF sig==MEMF
> THEN IF setbrk(brkincr) == -1
> THEN error(nospace);
> FI
> ELIF ...
> <endquote>
>
>
> This was already present in 7th edition, so it is by no means new in
32V or
SysIII -- it had just escaped my attention as a conceptual step in
the development of Unix memory handling.
>
4:38 a.m.
On Sat, Dec 31, 2022 at 07:55:28PM -0700, Warner Losh wrote:
On Sat, Dec 31, 2022 at 5:55 AM Paul Ruizendaal
<pnr(a)planet.nl> wrote:
"So DEC came to us at Holmdel. We were clearly the second string.
Tom London and John Reiser were interested and so was Ken Swanson, and
we got the VAX in early '78. I didn't do any of the technical work. In
fact, I devoted a lot of energy and time to getting the management to
let us do it. It wasn't research, you see. However, they let us take
the time. And in about three months my small group ported Version 7 to
the VAX. We got the machine in January, they had it running in April,
and by August it really worked.
...
So, with the blessings of BTL Area 11 management, we sent 32V to
Berkeley. It was in October or November, 1978"
Charlie Roberts in Salus QCU
"The operating system is Research version 7 as of April 15, 1978.
..
Work on the C compiler began in mid-December 1977. The hardware arrived
on March 3. We held a party on May 19 to celebrate successful multiuser
operation of the system."
London and Reiser
A UNIX Operating System for the DEC VAX-11/780 Computer
https://www.bell-labs.com/usr/dmr/www/otherports/32v.html
The "assembly code in the Bourne shell"
comment is in the same
London/Reiser paper. The full quote is:
"The (Bourne) shell is the standard user command interpreter. It required
by far the largest conversion effort of any supposedly portable program,
for the simple reason that it is not portable. Critical portions are coded
in assembly language and had to be painstakingly rewritten. The shell uses
its own sbrk which is functionally different from the standard routine in
libc. The shell wants the routine which fields a signal to be passed a
parameter giving the number of the signal being caught; signal was also a
private rou- tine. This was handled by having the operating system provide
the parameter in the first place, doing away with the private code for
signal. The code in fixargs (for constructing the argument list to an exec
system call) had to be diddled."
The files in the V7 tree on the Tuhs website are dated January 1979, so it
would seem that the fixes for 32V were immediately taken back to Research.
As you point out, this means that the comments above do not refer to the
well known source code, but to a predecessor of that (which I don’t think
survived).
We have ample evidence that V7 was really something more akin to a rolling
release. Let me explain: We know from the leaked '50 changes' tape that
many of the features were set earlier rather than later. This leaked in
1978 (if my notes are right), but I found references to it from as early as
November 1976 in
http://www.toad.com/early-usenix-newsletters/197611-unix-news-n11.pdf. This
was 18 months after V6 was released, but over 2 years before V7 was
released. In addition, we know from the AUUS newsletters in the archive
document that the V7 release process process took a while to get through
AT&T's legal department (IIRC a year, but I've not gone back to the AUUS
newsletters to refresh my recollection). A big push of V7 was to make it
portable as well (with AT&T doing an Interdata 8/32 port themselves, as
well as at least looking at the Wollongong Interdata 7/32 port and the
Harvard VM/370 port). In talking to Kirk and others that have been around
from approximately that time, 32V was widely viewed as V7 for Vaxen. We can
see evidence in the surviving 32V files of evolution from the 'PDP-11-like
swapping to a more sophisticated paging algorithm' since we have the
slowsys directory. It's my contention, as someone that coded in the era
before good source code control, that it's evidence that somebody got it
working, then renamed/copied it to slowsys while they got paging working so
they could build either kernel for A/B testing. Kirk has also told me that
the 32V port was started well in advance of V7's release to be both a
useful product inside of Bell Labs (since Vaxen were starting to appear) as
well as to prove that V7 was portable enough. I'll be the first to admit
this is at best conjecture that matches available facts, artifacts and old
timers recollections (sorry Kirk), but that we have no direct evidence for.
It also allowed the 3BSD efforts to get going before the official V7
release due to the close ties between Bell Labs and Berkeley and the DARPA
project around Unix.
I believe that we can conclude that the original 'hard to port' Bourne
shell was produced around the time of the 50 changes tape, give or take.
And that all the unix porting efforts that pre-dated the V7 release rolled
what appeared in 32V into V7 to reduce the amount of pdp-11 assembler. And
those efforts are what we read about in the paper.
"The Bourne shell work started either in early 1976, or maybe
late 1975. The first version was VERY different"
John Mashey in net.unix-wizards, 18 Mar 86
https://groups.google.com/g/alt.folklore.computers/c/xW3ZgEnFoFs
tuhs/Documentation/AUUGN/AUUGN-V06.6.pdf pg 39
"In 1976, Steve Bourne, who had recently joined 1127, wrote a new shell"
Kernighan, UNIX A History and a Memoir, 5.1 pg 88
Bourne's AsiaBSDCon 2016 talk also lists 1976
and goes on to discuss sbrk() use causing problems with ports
https://youtu.be/7tQ2ftt3LO8?t=715
5:25 a.m.
On Sat, Dec 31, 2022, 9:38 PM Jonathan Gray <jsg(a)jsg.id.au> wrote:
On Sat, Dec 31, 2022 at 07:55:28PM -0700, Warner Losh
wrote:
rolling
And at 5:18 he says he had a vax lab with three vaxen and the Lab's vax
port didn't have virtual memory. Bill Joy with 3BSD which had virtual
memory. They installed it on the vaxen because they were hitting physical
memory limits for some of their programs....
This suggests that the 32V in the TUHS archive is a later one than this
early port. 3BSD is listed as being released end of 1979...
Still begs the question of when 32v started paging. :)
But the time lines match up for the 32v effort to be fed back into the
research code base...
Warner
On Sat, Dec 31, 2022 at 5:55 AM Paul Ruizendaal
<pnr(a)planet.nl> wrote:
> The "assembly code in the Bourne shell" comment is in the same
> London/Reiser paper. The full quote is:
>
> "The (Bourne) shell is the standard user command interpreter. It
required
> by far the largest conversion effort of any
supposedly portable
program,
> for the simple reason that it is not
portable. Critical portions are
coded
> in assembly language and had to be
painstakingly rewritten. The shell
uses
> its own sbrk which is functionally different
from the standard routine
in
> libc. The shell wants the routine which
fields a signal to be passed a
> parameter giving the number of the signal being caught; signal was
also a
> private rou- tine. This was handled by
having the operating system
provide
> the parameter in the first place, doing away
with the private code for
> signal. The code in fixargs (for constructing the argument list to an
exec
> system call) had to be diddled."
>
> The files in the V7 tree on the Tuhs website are dated January 1979,
so it
> would seem that the fixes for 32V were
immediately taken back to
Research.
> As you point out, this means that the
comments above do not refer to
the
> well known source code, but to a predecessor
of that (which I don’t
think
survived).
We have ample evidence that V7 was really something more akin to a release. Let me explain: We know from the leaked
'50 changes' tape that
many of the features were set earlier rather than later. This leaked in
1978 (if my notes are right), but I found references to it from as early
as
November 1976 in
http://www.toad.com/early-usenix-newsletters/197611-unix-news-n11.pdf.
This
was 18 months after V6 was released, but over 2
years before V7 was
released. In addition, we know from the AUUS newsletters in the archive
document that the V7 release process process took a while to get through
AT&T's legal department (IIRC a year, but I've not gone back to the AUUS
newsletters to refresh my recollection). A big push of V7 was to make it
portable as well (with AT&T doing an Interdata 8/32 port themselves, as
well as at least looking at the Wollongong Interdata 7/32 port and the
Harvard VM/370 port). In talking to Kirk and others that have been around
from approximately that time, 32V was widely viewed as V7 for Vaxen. We
can
see evidence in the surviving 32V files of
evolution from the
'PDP-11-like
swapping to a more sophisticated paging
algorithm' since we have the
slowsys directory. It's my contention, as someone that coded in the era
before good source code control, that it's evidence that somebody got it
working, then renamed/copied it to slowsys while they got paging working
so
they could build either kernel for A/B testing.
Kirk has also told me
that
the 32V port was started well in advance of
V7's release to be both a
useful product inside of Bell Labs (since Vaxen were starting to appear)
as
well as to prove that V7 was portable enough.
I'll be the first to admit
this is at best conjecture that matches available facts, artifacts and
old
timers recollections (sorry Kirk), but that we
have no direct evidence
for.
It also allowed the 3BSD efforts to get going
before the official V7
release due to the close ties between Bell Labs and Berkeley and the
DARPA
project around Unix.
"So DEC came to us at Holmdel. We were clearly the second string.
Tom London and John Reiser were interested and so was Ken Swanson, and
we got the VAX in early '78. I didn't do any of the technical work. In
fact, I devoted a lot of energy and time to getting the management to
let us do it. It wasn't research, you see. However, they let us take
the time. And in about three months my small group ported Version 7 to
the VAX. We got the machine in January, they had it running in April,
and by August it really worked.
...
So, with the blessings of BTL Area 11 management, we sent 32V to
Berkeley. It was in October or November, 1978"
Charlie Roberts in Salus QCU
"The operating system is Research version 7 as of April 15, 1978.
..
Work on the C compiler began in mid-December 1977. The hardware arrived
on March 3. We held a party on May 19 to celebrate successful multiuser
operation of the system."
London and Reiser
A UNIX Operating System for the DEC VAX-11/780 Computer
https://www.bell-labs.com/usr/dmr/www/otherports/32v.html
I believe that we can conclude that the original 'hard to port' Bourne
shell was produced around the time of the 50 changes tape, give or take.
And that all the unix porting efforts that pre-dated the V7 release
rolled
what appeared in 32V into V7 to reduce the amount
of pdp-11 assembler.
And
those efforts are what we read about in the
paper.
"The Bourne shell work started either in early 1976, or maybe
late 1975. The first version was VERY different"
John Mashey in net.unix-wizards, 18 Mar 86
https://groups.google.com/g/alt.folklore.computers/c/xW3ZgEnFoFs
tuhs/Documentation/AUUGN/AUUGN-V06.6.pdf pg 39
"In 1976, Steve Bourne, who had recently joined 1127, wrote a new shell"
Kernighan, UNIX A History and a Memoir, 5.1 pg 88
Bourne's AsiaBSDCon 2016 talk also lists 1976
and goes on to discuss sbrk() use causing problems with ports
https://youtu.be/7tQ2ftt3LO8?t=715
5:35 a.m.
On Sun, Jan 1, 2023 at 12:27 AM Warner Losh <imp(a)bsdimp.com> wrote:
On Sat, Dec 31, 2022, 9:38 PM Jonathan Gray
<jsg(a)jsg.id.au> wrote:
One wonders what is meant by "virtual memory" in this context. I
contend that Unix has had "virtual memory" since moving off of the
PDP-11/20, in the sense of having a virtual address space that was
mapped onto a (possibly contiguous) physical address space. I think
all of these references mean demand paging, possibly with page
reclamation or whole-process swapping under memory pressure.
- Dan C.
[snip]
Bourne's AsiaBSDCon 2016 talk also lists 1976
and goes on to discuss sbrk() use causing problems with ports
https://youtu.be/7tQ2ftt3LO8?t=715
And at 5:18 he says he had a vax lab with three vaxen and the Lab's vax port
didn't have virtual memory. Bill Joy with 3BSD which had virtual memory. They
installed it on the vaxen because they were hitting physical memory limits for some of
their programs....
5:52 a.m.
At 2023-01-01T00:35:12-0500, Dan Cross wrote:
On Sun, Jan 1, 2023 at 12:27 AM Warner Losh
<imp(a)bsdimp.com> wrote:
I apologize if this point is too elementary, but I speculate that one
possible source of confusion comes from a file naming convention: which
of these (multiple) virtual memory or demand-paged VM systems installed
the kernel under the name "vmunix" vs. "unix".
Which ones did and did not?
When I was first learning Unix I asked a local expert why the kernel was
named "vmunix". They told me that it was because it supported virtual
memory (and explained what that was, because I was even more callow then
than now).
Then I asked where the non-VM kernel was. I was informed that there
wasn't one--it didn't even exist for modern architectures. I wondered
then why, if virtual memory was a given, you wouldn't just go back to
using the filename "unix".
I wondered similar things when encountering the "vmlinux" file a couple
of years later.
Reflexive obeisance to traditions has a cost.
Regards,
Branden
On Sat, Dec 31, 2022, 9:38 PM Jonathan Gray
<jsg(a)jsg.id.au> wrote:
One wonders what is meant by "virtual memory" in this context. I
contend that Unix has had "virtual memory" since moving off of the
PDP-11/20, in the sense of having a virtual address space that was
mapped onto a (possibly contiguous) physical address space. I think
all of these references mean demand paging, possibly with page
reclamation or whole-process swapping under memory pressure. [snip]
Bourne's AsiaBSDCon 2016 talk also lists 1976
and goes on to discuss sbrk() use causing problems with ports
https://youtu.be/7tQ2ftt3LO8?t=715
And at 5:18 he says he had a vax lab with three vaxen and the Lab's
vax port didn't have virtual memory. Bill Joy with 3BSD which had
virtual memory. They installed it on the vaxen because they were
hitting physical memory limits for some of their programs....
6:35 a.m.
On Sat, Dec 31, 2022, 10:53 PM G. Branden Robinson <
g.branden.robinson(a)gmail.com> wrote:
Yea... There was a thread in FreeBSD about what to name the kernel. Vmduck
was suggested as a joke...
Warner
> Regards,
> Branden
At 2023-01-01T00:35:12-0500, Dan Cross wrote:
3BSD started the vmunix name because it added virtual memory to unix
(paging aspect of virtual memory).
When I was first learning Unix I asked a local expert why the kernel was
On Sun, Jan 1, 2023 at 12:27 AM Warner Losh
<imp(a)bsdimp.com> wrote:
I apologize if this point is too elementary, but I speculate that one
possible source of confusion comes from a file naming convention: which
of these (multiple) virtual memory or demand-paged VM systems installed
the kernel under the name "vmunix" vs. "unix".
Which ones did and did not?
On Sat, Dec 31, 2022, 9:38 PM Jonathan Gray
<jsg(a)jsg.id.au> wrote:
One wonders what is meant by "virtual memory" in this context. I
contend that Unix has had "virtual memory" since moving off of the
PDP-11/20, in the sense of having a virtual address space that was
mapped onto a (possibly contiguous) physical address space. I think
all of these references mean demand paging, possibly with page
reclamation or whole-process swapping under memory pressure. [snip]
Bourne's AsiaBSDCon 2016 talk also lists 1976
and goes on to discuss sbrk() use causing problems with ports
https://youtu.be/7tQ2ftt3LO8?t=715
And at 5:18 he says he had a vax lab with three vaxen and the Lab's
vax port didn't have virtual memory. Bill Joy with 3BSD which had
virtual memory. They installed it on the vaxen because they were
hitting physical memory limits for some of their programs.... named "vmunix". They told me that it was
because it supported virtual
memory (and explained what that was, because I was even more callow then
than now).
Then I asked where the non-VM kernel was. I was informed that there
wasn't one--it didn't even exist for modern architectures. I wondered
then why, if virtual memory was a given, you wouldn't just go back to
using the filename "unix".
V7 had "unix" for sure. 32V still had unix if my quick grep is right.
I wondered similar things when encountering the "vmlinux" file a couple
of years later.
It was emulation of BSD and/or SunOS...
Reflexive obeisance to traditions has a cost.
6:35 a.m.
I distinctly remember I renamed to boot image on our research VAX (alice!)
from "vmunix" to "unix". It had been long enough by that point that
the
lingering prefix was starting to sound like a brag.
-rob
On Sun, Jan 1, 2023 at 4:53 PM G. Branden Robinson <
g.branden.robinson(a)gmail.com> wrote:
At 2023-01-01T00:35:12-0500, Dan Cross wrote:
On Sun, Jan 1, 2023 at 12:27 AM Warner Losh
<imp(a)bsdimp.com> wrote:
I apologize if this point is too elementary, but I speculate that one
possible source of confusion comes from a file naming convention: which
of these (multiple) virtual memory or demand-paged VM systems installed
the kernel under the name "vmunix" vs. "unix".
Which ones did and did not?
When I was first learning Unix I asked a local expert why the kernel was
named "vmunix". They told me that it was because it supported virtual
memory (and explained what that was, because I was even more callow then
than now).
Then I asked where the non-VM kernel was. I was informed that there
wasn't one--it didn't even exist for modern architectures. I wondered
then why, if virtual memory was a given, you wouldn't just go back to
using the filename "unix".
I wondered similar things when encountering the "vmlinux" file a couple
of years later.
Reflexive obeisance to traditions has a cost.
Regards,
Branden
On Sat, Dec 31, 2022, 9:38 PM Jonathan Gray
<jsg(a)jsg.id.au> wrote:
One wonders what is meant by "virtual memory" in this context. I
contend that Unix has had "virtual memory" since moving off of the
PDP-11/20, in the sense of having a virtual address space that was
mapped onto a (possibly contiguous) physical address space. I think
all of these references mean demand paging, possibly with page
reclamation or whole-process swapping under memory pressure. [snip]
Bourne's AsiaBSDCon 2016 talk also lists 1976
and goes on to discuss sbrk() use causing problems with ports
https://youtu.be/7tQ2ftt3LO8?t=715
And at 5:18 he says he had a vax lab with three vaxen and the Lab's
vax port didn't have virtual memory. Bill Joy with 3BSD which had
virtual memory. They installed it on the vaxen because they were
hitting physical memory limits for some of their programs....
6:27 a.m.
On Sat, Dec 31, 2022, 10:35 PM Dan Cross <crossd(a)gmail.com> wrote:
On Sun, Jan 1, 2023 at 12:27 AM Warner Losh
<imp(a)bsdimp.com> wrote:
It's clear from context it's demand paging and that unlocks processes
larger than physical memory. Imho.
Warner
On Sat, Dec 31, 2022, 9:38 PM Jonathan Gray
<jsg(a)jsg.id.au> wrote:
port
didn't have virtual memory. Bill Joy with 3BSD which had virtual
memory. They installed it on the vaxen because they were hitting physical
memory limits for some of their programs....
One wonders what is meant by "virtual memory" in this context. I
contend that Unix has had "virtual memory" since moving off of the
PDP-11/20, in the sense of having a virtual address space that was
mapped onto a (possibly contiguous) physical address space. I think
all of these references mean demand paging, possibly with page
reclamation or whole-process swapping under memory pressure.
[snip]
Bourne's AsiaBSDCon 2016 talk also lists 1976
and goes on to discuss sbrk() use causing problems with ports
https://youtu.be/7tQ2ftt3LO8?t=715
And at 5:18 he says he had a vax lab with three vaxen and the Lab's vax
2:50 p.m.
Paging was what they were referring to. The Bell releases at the time
only swapped like they did on the PDP-11.
BSD allowed only fragments of the program to need be resident.
The joke as “It ain’t virtual unless it isn’t all there."
------ Original Message ------
From "Dan Cross" <crossd(a)gmail.com>
To "Warner Losh" <imp(a)bsdimp.com>
Cc "Jonathan Gray" <jsg(a)jsg.id.au>au>; "Paul Ruizendaal"
<pnr(a)planet.nl>nl>;
"The Eunuchs Hysterical Society" <tuhs(a)tuhs.org>rg>; "segaloco"
<segaloco(a)protonmail.com>
Date 1/1/2023 12:35:12 AM
Subject [TUHS] Re: A few comments on porting the Bourne shell
On Sun, Jan 1, 2023 at 12:27 AM Warner Losh
<imp(a)bsdimp.com> wrote:
On Sat, Dec 31, 2022, 9:38 PM Jonathan Gray
<jsg(a)jsg.id.au> wrote:
One wonders what is meant by "virtual memory" in this context. I
contend that Unix has had "virtual memory" since moving off of the
PDP-11/20, in the sense of having a virtual address space that was
mapped onto a (possibly contiguous) physical address space. I think
all of these references mean demand paging, possibly with page
reclamation or whole-process swapping under memory pressure.
- Dan C. [snip]
Bourne's AsiaBSDCon 2016 talk also lists 1976
and goes on to discuss sbrk() use causing problems with ports
https://youtu.be/7tQ2ftt3LO8?t=715
And at 5:18 he says he had a vax lab with three vaxen and the Lab's vax port
didn't have virtual memory. Bill Joy with 3BSD which had virtual memory. They
installed it on the vaxen because they were hitting physical memory limits for some of
their programs....
7:11 a.m.
On Sat, Dec 31, 2022 at 10:25:49PM -0700, Warner Losh wrote:
And at 5:18 he says he had a vax lab with three vaxen
and the Lab's vax
port didn't have virtual memory. Bill Joy with 3BSD which had virtual
memory. They installed it on the vaxen because they were hitting physical
memory limits for some of their programs....
This suggests that the 32V in the TUHS archive is a later one than this
early port. 3BSD is listed as being released end of 1979...
It was 4BSD and 1980 according to QCU:
"In 1980 it was Bill Joy who flew to New Jersey and, together with Steve
Bourne, put 4BSD up in Bell Labs. "We did it overnight," Joy told me.
"We decided to do it about 9 p.m.; people showed up in the morning and a
new os was running. It was pretty amazing."
7:21 a.m.
On Sun, Jan 1, 2023, 12:11 AM Jonathan Gray <jsg(a)jsg.id.au> wrote:
On Sat, Dec 31, 2022 at 10:25:49PM -0700, Warner Losh
wrote:
He said 3BSD in the video.... Interesting...
Warner
And at 5:18 he says he had a vax lab with three
vaxen and the Lab's vax
port didn't have virtual memory. Bill Joy with 3BSD which had virtual
memory. They installed it on the vaxen because they were hitting physical
memory limits for some of their programs....
This suggests that the 32V in the TUHS archive is a later one than this
early port. 3BSD is listed as being released end of 1979...
It was 4BSD and 1980 according to QCU:
"In 1980 it was Bill Joy who flew to New Jersey and, together with Steve
Bourne, put 4BSD up in Bell Labs. "We did it overnight," Joy told me.
"We decided to do it about 9 p.m.; people showed up in the morning and a
new os was running. It was pretty amazing."
10:25 a.m.
On 1 Jan 2023, at 06:25, Warner Losh
<imp(a)bsdimp.com> wrote:
Still begs the question of when 32v started paging. :)
The London/Reiser paper includes:
"Work on the C compiler began in mid-December 1977. The hardware arrived on March 3.
We held a party on May 19 to celebrate successful multiuser operation of the
system."
John Reiser had the following timeline in a mail conversation about a year ago.
"Our VAX-11/780, the first one delivered to a customer, arrived Feb.12, 1978
(Lincoln's Birthday). Tom and I had been preparing using PDP-11/45 since
December, and we achieved "login: " on the console DECwriter by April 15
(the deadline for US income tax filing). The rest of 1978 was "tuning",
and preparing for the release of "UNIX-32/V" to UC Berkeley.”
The dates don’t match to the day, but the timeline is essentially the same (moreover,
delivery of VAX hardware is not the same as it being operational, achieving login: is but
the first step of bringing the user land up).
This sets a minimum timeline for arriving at a first version of “slowsys”. The timestamps
on the surviving files of slowsys show January 1979.
Joy and Babaoglu write in “Converting a Swap-Based System to do Paging in an Architecture
Lacking Page-Referenced Bits”:
"In the fall of 1978 the Computer Science Division of the University of California at
Berkeley purchased a VAX-11/780 and arranged to run an early version of UNIX for the VAX
provided by Bell Laboratories under a cooperative research agreement. [...] A subsequent
version of the system was capable of loading processes into noncontiguous real memory
locations, called scatter loading , and was able to swap only portions of a process,
called partial swapping, as deemed necessary by the memory contention. This would become
the basis for the paging system development discussed in this paper.”
This places the earliest moment for scatter paging as the Fall of 1978. The timestamps of
the surviving files of sys show March 1979, which is the latest moment.
The for the timeline of 32V with true demand paging John Reiser recalled:
"My annual performance review in early 1979 said "Well done; but don't ever
do
it again” because it was not regarded as "Research”. So what did I do? I did
it again; this time, with demand paging. I designed and implemented mmap() based
on experience with PDP-10/Tenex PMAP page mapping system call. I fretted over
introducing the first UNIX system call with 6 arguments.”
John recalls that it was a 6 or 7 months effort, with the last few being a "long
tail" of decreasing attention. The would place this third version as completed around
the Fall of 1979. This is consistent with Rob Pike’s recollection that he saw it being
demonstrated early in 1980.
In 1980 Research had to chose how to progress in the field of memory management. The
Reiser system was cool, but not supported by management and the BSD VM system was seen as
good enough to use. 3BSD is from early 1980, 4BSD from late 1980. Likely the conflicting
recollections of Bill Joy are because it was a snapshot somewhere in between (plus the
general issues around recollection). In any case, the 1985 snapshot of 8th edition appears
to have been based on 4.1BSD (early 1981), so that is when the systems split.
464
days inactive
476
days old
98 comments
31 participants
participants (31)
-
Adam Thornton -
arnold＠skeeve.com -
Bakul Shah -
Chet Ramey -
Chris Hanson -
Clem Cole -
Dan Cross -
Dave Horsfall -
G. Branden Robinson -
Greg 'groggy' Lehey -
John Cowan -
Jon Steinhart -
Jonathan Gray -
Joseph Holsten -
Larry McVoy -
Lars Brinkhoff -
Luther Johnson -
Marc Donner -
Marshall Conover -
Mary Ann Horton -
Paul Ruizendaal -
Paul Winalski -
Ralph Corderoy -
Rich Salz -
Rob Pike -
ron minnich -
Ron Natalie -
segaloco -
Steve Nickolas -
Sven Mascheck -
Warner Losh