TUHS July 2021

tuhs@tuhs.org

70 participants
40 discussions

The Unix shell: a 50-year view - hopefully final review of letter

by Jon Steinhart

Once again, thanks to everybody who as contributed to making this a better letter. Many of you have asked to be co-signers. Please let me know if I've included your name by mistake or if you'd like your name to be added. And, of course, let me know if any more edits are required. BTW, except where I'm quoting the paper I use UNIX instead of Unix as that's what I believe is the correct form. Please let me know if that's not correct. Thanks, Jon I read the "Unix Shell Programming: The Next 50 Years" paper expecting some well thought out wisdom. I was sorely disappointed. The paper is lacking the generally accepted form of: o What problem are you trying to solve? o What have others done? o What's our approach? o How does it do? Some particulars: o The paper never defines what is meant by the term "Unix shell." I think that you're using it to mean a command interpreter as described in the POSIX 1003.2 documents. o The paper makes liberal use of the term "Unix" such as "... in every Unix distribution." While systems descended from UNIX abound few actual instances of UNIX exist today. o There is no 50-year-old UNIX shell. I started using UNIX in the early 1970s, and the command interpreter at the time (Ken Thompson's shell) was nothing like later shells such as the Bourne shell (sh since research V7 UNIX), Korn shell (ksh), C shell (csh), and the Bourne again shell (bash). UNIX mainstreamed the notion of a command interpreter that was not baked into the system. The paper lacks any discussion of prior art. In practice, shell implementations either predate the POSIX standard or were built afterwards and include non-standard extensions. o The paper repeatedly claims that the shell has been largely ignored by academia and industry. Yet, it does not include any references to support that claim. In fact, the large body of published work on shells and ongoing work on shells such as zsh shows that claim to be incorrect. o The large number of pejorative statements detract from the academic value of the paper. And, in my opinion, these statements are provably false. It reads as if the authors are projecting their personal opinions onto the rest of the world. o The paper applies universal complaints such as "unmaintainable" to the shell; it doesn't call out any shell-specific problems. It doesn't explain whether these complaints are against scripts, implementations, or both. One of the reasons for the longevity of the family of shells descended from Bourne's sh is that experienced practitioners have been able to write easily maintainable code. Scripts written in the 1980s are still around and working fine. o The paper seems to complain about the fact that the shell is documented. This is astonishing. Proper documentation has always been a key component of being a professional, at least in my decades of experience. As a matter of fact, my boss telling me that "nobody will give a crap about your work unless you write a good paper" when I was a teenager at Bell Labs is what led me to UNIX and roff. o The paper includes non-sequiturs such as discussions about Docker and systemd that have nothing to to with the shell. o The paper has many "no-op" statements such as "arguably improved" that provide no useful information. o The example on page 105 don't work as there is no input to "cut". o The single result in Figure 1 is insufficient evidence that the approach works on a wide variety of problems. o The paper gives the appearance that the authors don't actually understand the Bourne shell semantics. Not just my opinion; Steve Bourne expressed that in an email to me after he read your paper, and I consider him to be a subject matter expert. o The paper confuses the performance of the shell with the performance of external commands executed by the shell. o Proofreading should have caught things like "improve performance performance" on page 107 among others. I think that the paper is really trying to say: o Programmable command interpreters such as those found in UNIX based systems have been around for a long time. For this paper, we're focusing on the GNU bash implementation of the POSIX P1003.2 shell. Other command interpreters predate UNIX. o This implementation is used more often than many other scripting languages because it is available and often installed as the default command interpreter on most modern systems (UNIX-based and otherwise). In particular, it is often the default for Linux systems. o The shell as defined above is being used in more complex environments than existed at the time of its creation. This exposes a new set of performance issues. o While much work has been done by the bash implementers, it's primarily been in the area of expanding the functionality, usually in a backward-compatible manner. Other shells such as the original ksh and later ash and zsh were implemented with an eye towards the performance of the internals and user perspectives. o Performance optimization using modern techniques such as JIT compilation have been applied to other languages but not to POSIX shell implementations. This paper looks at doing that. It is unsurprising that techniques that have worked elsewhere work here too. It's hard to imagine that the application of this technique is all that's required for a 50-year life extension. The title of this paper implies that it's going to be comprehensive but instead concentrates on a couple of projects. It ignores other active work on shells such as "fish". While it wouldn't eliminate the issues with the paper, they would not have been quite so glaring had it had a more modest title such as "Improving POSIX Shell Performance with JIT Compilation". Jon Steinhart plus John Cowan, Warner Losh, John Dow, Steve Bourne, Larry McVoy, and Clem Cole

1 year, 8 months

The Unix shell: a 50-year view -- feedback wanted

by Jon Steinhart

I not only found this paper offensive, but was more offended that ACM would publish something like this and give it an award to boot. I'm planning to send the authors and ACM what's below. Would appreciate any feedback that you could provide to make it better. Thanks, Jon I read your "Unix Shell Programming: The Next 50 Years" expecting some well thought out wisdom from learned experiences. I was sorely disappointed. o The paper never defines what is meant by the term "Unix shell." I think that you're using to mean a command interpreter as described in the POSIX 1003.2 documents. o There is no 50 year old Unix shell. I started using Unix in the early 1970s, and the command interpreter at the time (Ken Thompson's shell) was nothing like later shells such as the Bourne shell (sh since research V7 Unix), Korn shell (ksh), C shell (csh), and the Bourne again shell (bash). The paper is missing any discussion of prior art. In practice, shell implementations either predate the POSIX standard or were built afterwards and include non-standard extensions. o The paper repeats a claim that the shell has been largely ignored by academia and industry. Yet, it does not include any references that support that claim. My own experience and thus opinion is the opposite making the veracity of your claim questionable. As a reader, such unsubstantiated claims make me treat the entire content as suspect. o The paper applies universal complaints such as "unmaintainable" to the shell; it doesn't call out any shell-specific problems. It doesn't explain whether these complaints are against the scripts, the implementation, or both. One of the reasons for the longevity of the sh/bash shells is that experienced practicioners have been able to write easily maintainable code. Scripts written in the 1980s are still around and working fine. o The paper seems to complain that the fact that the shell is documented is a problem. This is an astonishing statement. In my decades as an acting professional, teacher, and author, proper documentation is a key component of being a professional. o The paper is full of non-sequiturs such as discussions about Docker and systemd that have nothing to to with the shell. o The paper has many "nop" statements such as "arguably improved" that don't actually say anything. o Examples, such as the one on page 105 don't work. o Proofreading should have caught things like "improve performance performance" on page 107 among others. o The references contain many more items than the paper actually references. Did you plagerize the bibliography and forget to edit it? o The single result in Figure 1 is insufficient evidence that the approach works on a wide variety of problems. o The paper makes it appear that the authors don't actually understand the semantics of the original Bourne shell. Not just my opinion; I received an email from Steve Bourne after he read your paper, and I consider him to be a subject matter expert. The paper is lacking the generally accepted form of: o What problem are you trying to solve? o What have others done? o What's our approach? o How does it do? Filtering out all of the jargon added for buzzword compliance, I think that the paper is really trying to say: o Programmable command interpreters such as those found in Unix-based systems have been around for a long time. For this paper, we're focusing on the GNU bash implementation of the POSIX P1003.2 shell. Other command interpreters predate Unix. o This implementation is used more often than many other scripting languages because it is available and often installed as the default command interpreter on most modern systems (Unix-based or otherwise). In particular, it is often the default for Linux systems. o The shell as defined above is being used in ways that are far more complex than originally contemplated when Bourne created the original syntax and semantics, much less the new features added by the POSIX standards committee. The combination of both the POSIX and bash extensions to the Bourne shell exposes a new set of limitations and issues such as performance. o While much work has been done by the bash implementors, it's primarily been in the area of expanding the functionality, usually in a backward-compatible manner. Other shells such as the original ksh and later ash and zsh were implemented with an eye towards the performance of the internals and user perspectives. o Performance optimization using modern techniques such as JIT have been applied to other languages but not to POSIX shell implementations. This paper looks at doing that. It is unsurprising that techniques that have worked elsewhere work here too. o It's hard to imagine that the application of this technique is all that's required for a 50-year life extension. The title of this paper implies that it's going to be comprehensive rather than just being a shameless plus for an author's project. Of course, this doesn't make much of a paper. Guessing that that's why it was so "bulked up" with irrelevancies. It appears that all of you are in academia. I can't imagine that a paper like this would pass muster in front of any thesis committee, much less get that far. Not only for content, but for lack of proofreading and editing. The fact that the ACM would publish such a paper eliminates any regret that I may have had in dropping my ACM membership.

1 year, 8 months

The Unix shell: a 50-year view -- second pass feedback wanted

by Jon Steinhart

Thanks to everyone who provided me feedback on the first pass, especially those who suggested "shopt -u flame-off". Here's the second version. Once again, would appreciate feedback. Thanks, Jon I read your "Unix Shell Programming: The Next 50 Years" expecting some well thought out wisdom. I was sorely disappointed. The paper is lacking the generally accepted form of: o What problem are you trying to solve? o What have others done? o What's our approach? o How does it do? Some particulars: o The paper never defines what is meant by the term "Unix shell." I think that you're using it to mean a command interpreter as described in the POSIX 1003.2 documents. o There is no 50 year old Unix shell. I started using Unix in the early 1970s, and the command interpreter at the time (Ken Thompson's shell) was nothing like later shells such as the Bourne shell (sh since research V7 Unix), Korn shell (ksh), C shell (csh), and the Bourne again shell (bash). Unix mainstreamed the notion of a command interpreter that was not baked into the system. The paper lacks any discussion of prior art. In practice, shell implementations either predate the POSIX standard or were built afterwards and include non-standard extensions. o The paper repeats a claim that the shell has been largely ignored by academia and industry. Yet, it does not include any references that support that claim. In fact, the large body of published work on shells and ongoing work on shells such as zsh shows that claim to be incorrect. o The paper applies universal complaints such as "unmaintainable" to the shell; it doesn't call out any shell-specific problems. It doesn't explain whether these complaints are against the scripts, the implementation, or both. One of the reasons for the longevity of the family of shells descended from Bourne's sh is that experienced practicioners have been able to write easily maintainable code. Scripts written in the 1980s are still around and working fine. o The paper seems to complain that the fact that the shell is documented is a problem. This is astonishing. Proper documentation has always been a key component of being a professional in my decades of experience. As a matter of fact, my boss telling me that "nobody will give a crap about your work unless you write a good paper" when I was a teenager at Bell Labs is what led me to UNIX and nroff. o The paper includes non-sequiturs such as discussions about Docker and systemd that have nothing to to with the shell. o The paper has many "nop" statements such as "arguably improved" that don't actually say anything. o Examples, such as the one on page 105 don't work as there is no input to "cut". o The single result in Figure 1 is insufficient evidence that the approach works on a wide variety of problems. o The paper gives the appearance that the authors don't actually understand the semantics of the original Bourne shell. Not just my opinion; I received an email from Steve Bourne after he read your paper, and I consider him to be a subject matter expert. o Proofreading should have caught things like "improve performance performance" on page 107 among others. I think that the paper is really trying to say: o Programmable command interpreters such as those found in Unix-based systems have been around for a long time. For this paper, we're focusing on the GNU bash implementation of the POSIX P1003.2 shell. Other command interpreters predate Unix. o This implementation is used more often than many other scripting languages because it is available and often installed as the default command interpreter on most modern systems (Unix-based or otherwise). In particular, it is often the default for Linux systems. o The shell as defined above is being used in ways that are far more complex than originally contemplated when Bourne created the original syntax and semantics, much less the new features from kash adopted by the POSIX standards committee. The combination of both the POSIX and bash extensions to the Bourne shell exposes a new set of limitations and issues such as performance. o While much work has been done by the bash implementors, it's primarily been in the area of expanding the functionality, usually in a backward-compatible manner. Other shells such as the original ksh and later ash and zsh were implemented with an eye towards the performance of the internals and user perspectives. o Performance optimization using modern techniques such as JIT compilation have been applied to other languages but not to POSIX shell implementations. This paper looks at doing that. It is unsurprising that techniques that have worked elsewhere work here too. It's hard to imagine that the application of this technique is all that's required for a 50-year life extension. The title of this paper implies that it's going to be comprehensive but instead concentrates on a couple of projects. It ignores other active work on shells such as "fish". While the issues with the paper remain, they would not have been quite so glaring had it had a more modest title such as "Applying JIT Compilation to the POSIX Shell".

1 year, 8 months

Good News:

by Caipenghui

UNIX history memoir has been published in China (publish by posts and Telecommunications Press in China) and translated into Chinese. It is also called name "UNIX legend" in China. I'm going to buy now. If you're in the computer industry, it's exciting just to understand how these buzzwords came into being. Even if you don't have a deep technical background, You can also benefit a lot from these ides that shine with genius. I have been looking forward to a Chinese book about the history of Unix development for so many years, and now I can finally see it. Thanks bwk and other guys.

1 year, 8 months

Re: [TUHS] Binary log files

by norman＠oclsc.org

Some of us have, literally for decades, been dealing with wtmp by rolling it weekly or monthly or quarterly or whatever, letting cron run something like cd /usr/adm # that's how long I've been doing this! umask 022 >wtmp.new ln wtmp wtmp.prev mv wtmp.new wtmp # also so long ago there was no seq(1) nums=`awk 'BEGIN {for (i = 12; i >= 0; i--) print i; exit}'` for i in $nums; do inext=`expr $i + 1` if [ -f wtmp.$i ]; then mv wtmp.$i wtmp.$inext fi done mv wtmp.prev wtmp.0 This really isn't rocket science. It isn't even particularly interesting UNIX history. Can we move on to something that IS interesting? Here are some things I find more interesting: 1. utmp came before wtmp: utmp(V) appears in the First Edition manual, wtmp(V) only in the Second. Apparently interest in who else is logged in right now predated interest in who has logged in recently. 2. Both files started out in /tmp. wtmp is first said to be in /usr/adm instead in the Fifth Edition manual, utmp in /etc in the Sixth. 3. The names /tmp/utmp and /tmp/wtmp appear to have been issued by the Department of Redundancy Department. I think it quite likely that Ken and Dennis would have been familiar with that joke once the recording containing it was issued in mid-1970, but I don't know whether utmp existed in some form before that. I see no sign of it in the fragments of PDP-7 source code we have (in particular init doesn't seem to use it), but what about later PDP-7 or very early PDP-11 code predating the late-1971 First Edition manual? Norman Wilson Toronto ON Not Insane

1 year, 8 months

Binary log files

by John Cowan

As long ago as the 7th Edition, several binary log files were maintained: the file generated by acct(2) (one record per process) and the utmp and wtmp files (one record per login). Both of these are defined by structs in .h files, so they are definitely not portable (int sizes, endianism, etc.)

1 year, 8 months

[tuhs] Dennis Ritchie's couch

by Nelson H. F. Beebe

The last article of the latest issue of the Communications of the ACM that appeared electronically earlier today is a brief interview with this year's ACM Turing Award winners, Al Aho and Jeff Ullman. The article is Last byte: Shaping the foundations of programming languages https://doi.org/10.1145/3460442 Comm. ACM 64(6), 120, 119, June 2021. and it includes a picture of the two winners sitting on Dennis Ritchie's couch. I liked this snippet from Jeff Ullman, praising fellow list member Steve Johnson's landmark program, yacc: >> ... >> At the time of the first Fortran compiler, it took several >> person-years to write a parser. By the time yacc came around, >> you could do it in an afternoon. >> ... ------------------------------------------------------------------------------- - Nelson H. F. Beebe Tel: +1 801 581 5254 - - University of Utah FAX: +1 801 581 4148 - - Department of Mathematics, 110 LCB Internet e-mail: beebe(a)math.utah.edu - - 155 S 1400 E RM 233 beebe(a)acm.org beebe(a)computer.org - - Salt Lake City, UT 84112-0090, USA URL: http://www.math.utah.edu/~beebe/ - -------------------------------------------------------------------------------

1 year, 8 months

Re: [TUHS] The Unix shell: a 50-year view

by Jon Steinhart

Wow. This is a terrible paper. It's full of of incorrect, unsubstiantiated, and meaningless statements. It's so bad in my opinion that I'm not sorry that I dropped my ACM membership a while ago. These folks really needed an editor. The paper annoys me so much that I've decided to write a lengthy note to the authors which I'll post here once I'm done. Jon

1 year, 8 months

Disassemblers

by Henry Bent

>From what I can gather the only way to reasonably examine the disassembly of a program in the early days of Unix was adb. Is this true? Was there a way to easily produce a full disassembly? I'll confess to being fairly ignorant of adb use since I always had dbx or the equivalent available. The first tool I'm aware of to purposefully/primarily produce a full listing is MIPS dis (ca. 1986?) but there must have been something before that for other systems, no? -Henry

1 year, 8 months

pointer disambiguation (was Re: Disassemers)

by Paul Winalski

On 7/1/21, scj(a)yaccman.com <scj(a)yaccman.com> wrote: > When PCC came along and started running on 32-bit machines, I started > thinking about algorithms for optimization. A problem that I had no > good solution for could be illustrated by a simple piece of code: > > x = *p; > > y = *q; > > q gets changed > > *q = z; > > The question is, do I need to reload x now because q might have been > changed to point to the same place as p? Yes, this is a very well-known problem in scalar optimization in compiler engineering. It's called pointer disambiguation and is part of the more general problem of data flow analysis. As you observed, getting it wrong can lead to very subtle and hard-to-debug correctness problems. In the worst case, one has to throw out all current data flow analysis of global and currently active local variables and start over. In your example, the statement "*q = z" may end up forcing the compiler to toss out all data flow information on x and z (and maybe p and q as well). If q could possibly point to x and x is in a register, the assignment forces x to be reloaded before its next use. Ambiguous pointers prohibit a lot of important optimizations. This problem is the source of a lot of bugs in compilers that do aggressive optimizations. Fortunately a bit of knowledge of just how "q gets changed" can rescue the situation. In strongly-typed languages, for example, if x and z are different data types, we know the assignment of z through q can't affect x. We also know that the assignment can't affect x if x and z have disjoint scopes. The 'restrict' keyword in C pointer declarations was added to help mitigate this problem. Some compilers also have a command line option that allows the user to say, "I solemnly swear that I won't do this sort of thing". -Paul W.

1 year, 8 months

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

1996

1995

1994

1993

1992

1991

1990

TUHS July 2021