16 Jan
2011
16 Jan
'11
8:43 a.m.
Hi all, best wishes for 2011. I had an e-mail from Sven Mascheck asking about
the history of #! interpretation in System V. I couldn't find any #!
code in the kernels before SysVR4. However, I thought I'd pass the
query onto the TUHS list, in case others can shed some light on the question.
Did SysV systems before r4 do #! interpretation, and if so where was it done:
kernel, library, shell? Any code references, e.g. function names etc.?
Many thanks,
Warren
16 Jan
16 Jan
9:55 a.m.
Warren Toomey scripsit:
Hi all, best wishes for 2011. I had an e-mail from
Sven Mascheck asking about
the history of #! interpretation in System V. I couldn't find any #!
code in the kernels before SysVR4.
That's correct. The feature was added to the 8th Research edition kernel,
but didn't migrate to the System III/V line until SVR4. Shebangs began
very early in BSD as a csh-only hack, but appeared as a config option
in kernels as early as 2.8BSD, according to Wikipedia; 4.2BSD was the
first release that turned them on by default.
Perl has always (I think) had its own shebang support; you can get Perl
to exec an arbitrary interpreter for a script provided it has a shebang.
Modern shells will use /bin/sh to run scripts that *don't* have shebangs,
except for ksh which considers itself /bin/sh-compatible, and therefore
runs such scripts itself.
--
I am expressing my opinion. When my John Cowan
honorable and gallant friend is called, cowan(a)ccil.org
he will express his opinion. This is http://www.ccil.org/~cowan
the process which we call Debate. --Winston Churchill
10:53 a.m.
Hi Warren
I worked on SVR3 maintenance at Philips Information Systems in the dim past. I'm
almost certain, like 99%, that #! was not in R3
Wilko
Op 16 jan. 2011 om 09:43 heeft Warren Toomey <wkt(a)tuhs.org> het volgende
geschreven:
Hi all, best wishes for 2011. I had an e-mail from
Sven Mascheck asking about
the history of #! interpretation in System V. I couldn't find any #!
code in the kernels before SysVR4. However, I thought I'd pass the
query onto the TUHS list, in case others can shed some light on the question.
Did SysV systems before r4 do #! interpretation, and if so where was it done:
kernel, library, shell? Any code references, e.g. function names etc.?
Many thanks,
Warren
_______________________________________________
TUHS mailing list
TUHS(a)minnie.tuhs.org
https://minnie.tuhs.org/mailman/listinfo/tuhs
1:20 p.m.
John Cowan wrote:
Warren Toomey scripsit:
> the history of #! interpretation in System V. I
couldn't find any #!
> code in the kernels before SysVR4.
The feature was added to the 8th Research edition
kernel,
but didn't migrate to the System III/V line until SVR4.
Thanks to you all. Amazing, it came so "late".
Shebangs began very early in BSD as a csh-only hack,
but appeared
as a config option in kernels as early as 2.8BSD, according to
Wikipedia; 4.2BSD was the first release that turned them on by
default.
In a usenet discussion with G.Ritter in '01 I noticed the first detailed
hint about the origin, and since then I've tried my best to document
the issue on http://www.in-ulm.de/~mascheck/various/shebang/#origin
(And although someone added some of the links I collected,
I believe Wikipedia still has several things mixed up,
while I'm quite busy to keep my own stuff consistent...)
4.0BSD had incorporated it originally among the BSDs.
2.8BSD came soon, sounds earlier, but after its original release,
2BSD was just a different development line (http://www.levenez.com/unix/unix_a4.pdf).
As you mention "shebangs began .. as csh-only hack":
I still wonder if one may call the BSD csh-hack as origin,
because #! might have been developed independently at Bell Labs.
(although, in turn, the idea might origin from Berkeley, as the
interesting answer from DMR in the wikipedia article suggests.)
And BTW: it was not "csh-only", but sh was hacked symmetrically :)
Perl has always (I think) had its own shebang support;
you can get Perl
to exec an arbitrary interpreter for a script provided it has a shebang.
The traditional ash, bash-1 (and -2 ff. as compile time option) also
implement that.
Modern shells will use /bin/sh to run scripts that
*don't* have shebangs,
except for ksh which considers itself /bin/sh-compatible, and therefore
runs such scripts itself.
I thought sh-like shells in general try to interprete scripts themselves
upon an ENOEXEC. Do you know certain shells which actually call "/bin/sh"?
Sven
5:17 p.m.
Sven Mascheck scripsit:
As you mention "shebangs began .. as csh-only
hack":
I still wonder if one may call the BSD csh-hack as origin,
because #! might have been developed independently at Bell Labs.
No, that would be too much of a coincidence to choose exactly the
same characters. There has to be a single point of origin, from
which it spread by stimulus diffusion (you hear there are telescopes,
you know they use lenses, you build your own telescope).
I thought sh-like shells in general try to interprete
scripts themselves
upon an ENOEXEC. Do you know certain shells which actually call "/bin/sh"?
Yes, seemingly bash does that too. Bogus.
--
John Cowan cowan(a)ccil.org http://ccil.org/~cowan
The whole of Gaul is quartered into three halves.
--Julius Caesar
7:19 p.m.
John Cowan wrote:
How'd you get the impression? I can't verify this.
bash-1.05:execute_cmd.c executed the commands itself, in a subshell,
" [ errno == ENOEXEC ]
/* This file is executable.
If it begins with #!, then help out people with losing
operating systems. Otherwise, check to see if it is a binary
file by seeing if the first line (or upto 30 characters) are in
the ASCII set. Execute the contents as shell commands. */ "
and this hasn't been changed (except length or return status)
until the current release.
Perhaps you had another shell in mind? I believe a shell should
try to execute commands itself instead of calling any other shell.
(That's the only way how executing on in POSIX shell is possible, BTW,
because neither #! nor paths are standardized).
Sven Mascheck scripsit:
This doesn't sound obligatory to me (I know it's academic).
DMR might have known the csh-hack (and if a comment character is
implemented anywhere, here csh, then it's obvious to use it for #!)
As you mention "shebangs began .. as
csh-only hack":
I still wonder if one may call the BSD csh-hack as origin,
because #! might have been developed independently at Bell Labs.
No, that would be too much of a coincidence to choose exactly the
same characters. There has to be a single point of origin, from
which it spread by stimulus diffusion (you hear there are telescopes,
you know they use lenses, you build your own telescope). I thought
sh-like shells in general try to interprete scripts themselves
upon an ENOEXEC. Do you know certain shells which actually call "/bin/sh"?
Yes, seemingly bash does that too. Bogus.
8:17 p.m.
Sven Mascheck scripsit:
How'd you get the impression? I can't verify this.
No, I meant that bash does what ksh does: uses itself, which means that a
shebang-free script runs differently on bash, ksh, dash. This is IMHO
unfortunate.
DMR might have known the csh-hack (and if a comment
character is
implemented anywhere, here csh, then it's obvious to use it for #!)
That might account for the "#", but not for "#!" taken together.
Having two different people invent the shebang independently (as opposed
to *implementing* it independently, as has happened many times -- 8th
Edition, SVR4, Linux, etc.) is just too improbable for me to swallow.
I thought sh-like shells in general try to interprete
scripts themselves
upon an ENOEXEC. Do you know certain shells which actually call "/bin/sh"?
Yes, seemingly bash does that too. Bogus. (That's the only way how executing on in POSIX
shell is possible, BTW,
because neither #! nor paths are standardized).
Good point. IMHO the Posix sh definition should be extended so that all
shells claiming Posix-compliance should do shebangs.
--
John Cowan cowan(a)ccil.org http://ccil.org/~cowan
And now here I was, in a country where a right to say how the country should
be governed was restricted to six persons in each thousand of its population.
For the nine hundred and ninety-four to express dissatisfaction with the
regnant system and propose to change it, would have made the whole six
shudder as one man, it would have been so disloyal, so dishonorable, such
putrid black treason. --Mark Twain's Connecticut Yankee
8:42 p.m.
On Sun, Jan 16, 2011 at 03:17:46PM -0500, John Cowan wrote:
Sven Mascheck scripsit:
I grepped 32v, v7, sysIII for it and didn't find it.
--
---
Larry McVoy lm at bitmover.com http://www.bitkeeper.com
DMR might have known the csh-hack (and if a
comment character is
implemented anywhere, here csh, then it's obvious to use it for #!)
That might account for the "#", but not for "#!" taken together.
Having two different people invent the shebang independently (as opposed
to *implementing* it independently, as has happened many times -- 8th
Edition, SVR4, Linux, etc.) is just too improbable for me to swallow.
9:08 p.m.
John Cowan wrote:
who (or what system) do you mean was the 2nd?
Sven Mascheck scripsit:
DMR might have
known the csh-hack [...]
Having two different people invent the shebang independently > > > > Modern shells will use /bin/sh to
run scripts that *don't* have shebangs,
[...]
No, I meant that bash does what ksh does: uses itself,
[...]
With /bin/sh you actually meant any shell calling "itself"?
Otherwise I'm afraid, I have some difficulties following you.
IMHO the Posix sh definition should be extended so
that all
shells claiming Posix-compliance should do shebangs.
Shells themselves should implement it, not the kernel?
There was a working group resolution to standardize #!, which didn't
make it, http://www.opengroup.org/platform/resolutions/bwg2000-004.html
you could chime in
9:37 p.m.
Sven Mascheck scripsit:
It might be too much to ask the kernel to do, especially on
non-traditional Posix systems like z/OS.
who (or what system) do you mean was the 2nd?
That was hypothetical.
Trying again: "It's unlikely that two different people chose #! as the
executable-script mark independently."
With /bin/sh you actually meant any shell calling
"itself"?
Otherwise I'm afraid, I have some difficulties following you.
I mean that whereas (t)csh uses /bin/sh to run scripts with no
shebangs, the Posix-compatible shells execute scripts with no
shebangs directly. And this is Bad.
IMHO the Posix
sh definition should be extended so that all
shells claiming Posix-compliance should do shebangs.
Shells themselves should implement it, not the kernel? There was a working group resolution to standardize
#!, which didn't
make it, http://www.opengroup.org/platform/resolutions/bwg2000-004.html
you could chime in
I see the problem now: a portable awk script, for example, can't assume
that the Posix awk is in /usr/bin/awk, so "#!/usr/bin/awk" might get the
wrong awk. So the feature is not worth standardizing for Posix.
--
[W]hen I wrote it I was more than a little John Cowan
febrile with foodpoisoning from an antique carrot cowan(a)ccil.org
that I foolishly ate out of an illjudged faith http://ccil.org/~cowan
in the benignancy of vegetables. --And Rosta
17 Jan
17 Jan
10:09 a.m.
On 16 Jan 2011, at 20:17, John Cowan wrote:
That might account for the "#", but not for
"#!" taken together.
Having two different people invent the shebang independently (as opposed
to *implementing* it independently, as has happened many times -- 8th
Edition, SVR4, Linux, etc.) is just too improbable for me to swallow.
One possibility is that there was prior art for "!" meaning "run in a
shell" which gave various people the same idea independently: in a script it needs to
be commented out so the shell will not barf, so you pretty naturally get "#!" as
the magic sequence. That's really arguing that the inventions were not independent
but they were both copying from something else.
The obvious place that "!" might have come from is ed: it seems to have been
there essentially for ever (the 1st edition manuals have it).
--tim
4:25 p.m.
On 01/16/2011 03:53, Wilko Bulte wrote:
Hi Warren
I worked on SVR3 maintenance at Philips Information Systems in the dim past. I'm
almost certain, like 99%, that #! was not in R3
That matches my memory of various 3B1 machines I had to use as part of
my support activities at The Wollognong Group back in 1989. All the
SIII machines didn't do #!, while some of the SysV machines did. I do
know that r1 didn't have it, but r4 did.
Warner
Wilko
Op 16 jan. 2011 om 09:43 heeft Warren Toomey<wkt(a)tuhs.org> het volgende
geschreven:
Hi all, best wishes for 2011. I had an e-mail
from Sven Mascheck asking about
the history of #! interpretation in System V. I couldn't find any #!
code in the kernels before SysVR4. However, I thought I'd pass the
query onto the TUHS list, in case others can shed some light on the question.
Did SysV systems before r4 do #! interpretation, and if so where was it done:
kernel, library, shell? Any code references, e.g. function names etc.?
Many thanks,
Warren
_______________________________________________
TUHS mailing list
TUHS(a)minnie.tuhs.org
https://minnie.tuhs.org/mailman/listinfo/tuhs
_______________________________________________
TUHS mailing list
TUHS(a)minnie.tuhs.org
https://minnie.tuhs.org/mailman/listinfo/tuhs
4:35 p.m.
On 01/16/2011 14:37, John Cowan wrote:
Sven Mascheck scripsit:
The alternatives are worse.
On a sysIII system, you had the situation where scripts would be
executed by whatever your shell. So if you put a script on the system,
you had to make sure everybody used csh and write it in csh, or you had
to make sure nobody did and write it in /bin/sh.
The current situation says that 'you must opt-in' to a different shell,
which leads to more predictable behavior.
Later, #! processing moved into the kernel so that interpreters would
execute things directly.
To cope with that confusing mess, the perl env hack was born so that the
perl scripts could execute anywhere, independent of the shell driving it.
Warner
With /bin/sh you actually meant any shell calling
"itself"?
Otherwise I'm afraid, I have some difficulties following you.
I mean that
whereas (t)csh uses /bin/sh to run scripts with no
shebangs, the Posix-compatible shells execute scripts with no
shebangs directly. And this is Bad. 
7:02 p.m.
4BSD (4.0) usr/src/sys/sys/TODO (of Nov. 9, 1980) says it was planned:
6. Exec fixes
Implement dmr's #! feature; pass string arguments through
faster.
And the usr/src/sys/newsys/sys1.c has explanation of it and source code.
http://minnie.tuhs.org/cgi-bin/utree.pl?file=4BSD/usr/src/sys/sys/TODO
http://minnie.tuhs.org/cgi-bin/utree.pl?file=4BSD/usr/src/sys/newsys/sys1.c
This seems to indicate the idea was not developed separately at
Berkeley.
It was introduced into BSD by April 1981. For 2BSD (2.8) it was added by
Dec. 16, 1981 when built with MENLO_SCRIPT defined (but I don't see that
documented or defined).
8:47 p.m.
Warner Losh scripsit:
The alternatives are worse.
You're making my point for me.
On a sysIII system, you had the situation where
scripts would be
executed by whatever your shell. So if you put a script on the system,
you had to make sure everybody used csh and write it in csh, or you had
to make sure nobody did and write it in /bin/sh.
Right.
The current situation says that 'you must
opt-in' to a different shell,
which leads to more predictable behavior.
Unfortunately not. The current situation is that when a shell sees a
shebang-free script, it decides whether it is Posix enough to execute
that script. So ash, bash, dash, ksh, zsh will all run shebang-free
scripts themselves, whereas csh, tcsh will defer to /bin/sh.
What I am arguing for is that *all* shells should defer to /bin/sh
to execute shebang-free scripts. As things stand, the behavior of a
shebang-free script depends on which shell is trying to run it, whereas
it should only depend (says I) on the identity of /bin/sh.
Later, #! processing moved into the kernel so that
interpreters would
execute things directly.
Yes, and that's a Good Thing.
--
The experiences of the past show John Cowan
that there has always been a discrepancy cowan(a)ccil.org
between plans and performance. http://www.ccil.org/~cowan
--Emperor Hirohito, August 1945
8:51 p.m.
Jeremy C. Reed scripsit:
This seems to indicate the idea was not developed
separately at
Berkeley.
Thanks for doing the digging. What this shows is that dmr came up with
the idea of putting shebang processing into the kernel, but it does not
show where shebangs as a shell feature came from. I still think that
has to be the CSRG.
It was introduced into BSD by April 1981. For 2BSD
(2.8) it was added by
Dec. 16, 1981 when built with MENLO_SCRIPT defined (but I don't see that
documented or defined).
Thanks again for clarifying the ordering here: I had assumed that 2.8
was older than 4.0.
--
We call nothing profound cowan(a)ccil.org
that is not wittily expressed. John Cowan
--Northrop Frye (improved)
8:58 p.m.
Jeremy C. Reed wrote:
4BSD (4.0) usr/src/sys/sys/TODO (of Nov. 9, 1980)
says it was planned:
It was not until now that I noticed that the subject is _that_ general :)
I had already tried to completely collect all the interesting pieces
available through TUHS (and the CSRG archive) on a page as I mentioned
earlier in this thread. But I'd like to tie in with
For 2BSD (2.8) it was added by Dec. 16, 1981 when
built with
MENLO_SCRIPT defined (but I don't see that documented or defined).
According to /usr/kernel/src/cmd/standalone/tapes/instructions
these seem to be fixes from the U.S. Geological Survey in Menlo Park,
perhaps by Bill Jolitz. The code looks different from 4BSD, because
2.8BSD was a 7th ed kernel with several adds+fixes (all to be activated
by macros like the above). I don't know yet how #! was created/derived
here, exactly - which remindes me that I don't know how the original
implementation (research unix) looked exactly. I always had the impression
that 4.0BSD newsys/sys1.c already had been adjusted to BSD...
10:41 p.m.
On Mon, 17 Jan 2011, John Cowan wrote:
As a similar feature, Joy's experimental "ashell" in the first BSD had a
feature to run Pascal objects (started with the magic 0404 octal word)
with the px interpreter. In 2BSD (and then 3BSD), his new csh had the
OTHERSH feature -- if it started with # it would use csh; if the script
did not begin with the # it would use /bin/sh.
I still don't see the same #! feature, but the above appear they may be
inspiration. Also the 1BSD and 2BSD predate CSRG.
This seems to
indicate the idea was not developed separately at
Berkeley.
Thanks for doing the digging. What this shows is that dmr came up with
the idea of putting shebang processing into the kernel, but it does not
show where shebangs as a shell feature came from. I still think that
has to be the CSRG. It was
introduced into BSD by April 1981. For 2BSD (2.8) it was added by
Dec. 16, 1981 when built with MENLO_SCRIPT defined (but I don't see that
documented or defined).
Thanks again for clarifying the ordering here: I had assumed that 2.8
was older than 4.0. 
19 Jan
19 Jan
4:10 a.m.
Le 17/01/2011 20:02, Jeremy C. Reed a écrit :
4BSD (4.0) usr/src/sys/sys/TODO (of Nov. 9, 1980) says it was planned:
6. Exec fixes
Implement dmr's #! feature; pass string arguments through
faster.
And the usr/src/sys/newsys/sys1.c has explanation of it and source code.
http://minnie.tuhs.org/cgi-bin/utree.pl?file=4BSD/usr/src/sys/sys/TODO
http://minnie.tuhs.org/cgi-bin/utree.pl?file=4BSD/usr/src/sys/newsys/sys1.c
This seems to indicate the idea was not developed separately at
Berkeley.
It was introduced into BSD by April 1981. For 2BSD (2.8) it was added by
Dec. 16, 1981 when built with MENLO_SCRIPT defined (but I don't see that
documented or defined).
Hi,
yet another reference but more in the spirit of what csh does, it only
checks for a simple hash (#), no explaim mark (!), and is enclosed in
UCB_SCRIPT define.
http://minnie.tuhs.org/cgi-bin/utree.pl?file=2.9BSD/usr/src/sys/sys/sys1.c
2.11 BSD seems to have an enhanced version of this feature in the sense
where the shell path may be followed by some arguments (i.e.: /bin/sh -x)
http://minnie.tuhs.org/cgi-bin/utree.pl?file=2.11BSD/sys/sys/kern_exec.c
Regards,
Cyrille Lefevre
--
mailto:Cyrille.Lefevre-lists@laposte.net
8:35 p.m.
Cyrille Lefevre wrote:
yet another reference but more in the spirit of what
csh does, it only
checks for a simple hash (#), no explaim mark (!), and is enclosed in
UCB_SCRIPT define.
http://minnie.tuhs.org/cgi-bin/utree.pl?file=2.9BSD/usr/src/sys/sys/sys1.c
Perhaps you missed the ! in the macro?
#define SCRMAG '#!'
2.11 BSD seems to have an enhanced version of this
feature in the sense
where the shell path may be followed by some arguments (i.e.: /bin/sh -x)
Yes, #! originally had not implemented arguments at all (this even applies
to 386BSD). Arguments ("all in one") came with 4.2BSD and later, variations
appeared like splitting up into argv[] or delivering only the "first" argument.
--
http://www.in-ulm.de/~mascheck/various/shebang/#results
20 Jan
20 Jan
4:09 a.m.
Le 19/01/2011 21:35, Sven Mascheck a écrit :
Cyrille Lefevre wrote:
right %-/
yet another reference but more in the spirit of
what csh does, it only
checks for a simple hash (#), no explaim mark (!), and is enclosed in
UCB_SCRIPT define.
http://minnie.tuhs.org/cgi-bin/utree.pl?file=2.9BSD/usr/src/sys/sys/sys1.c
Perhaps you missed the ! in the macro? #define SCRMAG '#!'
multi-character character constant !
u.u_exdata.ux_mag == SCRMAG makes me wonder...
$ cat x.c
#include <stdio.h>
main() { int c = '#!'; printf("%x", c); return 0; }
$ make x
cc x.c -o x
x.c:2:18: warning: multi-character character constant
$ ./x
2321
$ man ascii
...
| 20 sp | 21 ! | 22 " | 23 # | 24 $ | 25 % | 26 & | 27 ' |
...
well, it works !
Regards,
Cyrille Lefevre
--
mailto:Cyrille.Lefevre-lists@laposte.net
28 Jan
28 Jan
7:38 p.m.
On Mon, 17 Jan 2011, Jeremy C. Reed wrote:
It was introduced into BSD by April 1981. For 2BSD
(2.8) it was added by
Dec. 16, 1981 when built with MENLO_SCRIPT defined (but I don't see that
documented or defined).
Doing some related research, I noticed I had written in my notes (maybe
over a year ago) that Robert Elz added #! to BSD. Here is the email
message with the background:
http://mail-index.netbsd.org/netbsd-users/2008/11/10/msg002390.html
30 Jan
30 Jan
3:05 p.m.
Jeremy C. Reed wrote:
Doing some related research, I noticed I had written in my notes (maybe
over a year ago) that Robert Elz added #! to BSD. Here is the email
message with the background:
http://mail-index.netbsd.org/netbsd-users/2008/11/10/msg002390.html
Thanks for digging that out. I was confused at first, because 4.0 BSD
implemented it without arguments, while the above article tells
"When I added it to BSD, I made a conscious decision to permit
exactly one arg - not a general command line."
But the first regular occurence (active by default) in 4.2BSD
is implemented like that.
Do you feel like providing access to more of your research?
4837
days inactive
4851
days old
22 comments
9 participants
participants (9)
-
Cyrille Lefevre -
Jeremy C. Reed -
John Cowan -
Larry McVoy -
Sven Mascheck -
Tim Bradshaw -
Warner Losh -
Warren Toomey -
Wilko Bulte