3 Jan
2023
3 Jan
'23
3:08 p.m.
segaloco via TUHS writes:
> I think that's a good point that scripting problems may be
> a symptom of the nature of the tools being used in them.
I think that you're hinting at something
different.
To the best of my recollection, scripting languages
were originally
intended and used for the automation of repetitive personal tasks;
making it easier for users who found themselves typing the same
stuff over and over again.
Indeed!
Somewhere along the line people forgot
how to use a compiler and began writing large systems in a variety
of roughly equivalent but incompatible interpreted languages. Can
one even boot linux without having several different incompatible
versions of Python installed today? So I don't think that it's the
nature of the tools; I think that it's people choosing the wrong
tools for the problems that they're trying to solve.
Jon
The forgotten compilers were typically used to supply glue
to paste major tools together. The nature of that glue---often
simple data reformatting--inspired tools like sed and awk.
Each use of a tool became a process that saved many minutes
of work that would in a scriptless world be guided by hand,
boringly and unreliably.
Yet glue processes typically did only microseconds of
"real" work. In the name of efficiency, the operations began
to be incorporated directly into the shell. The first
inklings of this can be seen in "echo" and various forms
of variable-substitution making their way into the v7
shell. The phenomenon proliferated into putting what were
typically canned sed one-liners (but not sed itself) into
the shell.
Lots of specializations crowded out universality. A side
effect was an explosion of knowledge required to write
or understand code. Such is the tragedy of "forgetting
compilers".
Doug
3 Jan
3 Jan
3:57 p.m.
New subject: A few comments on porting the Bourne shell
Hi Doug!
On 1/3/23 16:08, Douglas McIlroy wrote:
Well, as time goes by, I'm also writing less and less programs. But only
because I find that I can pipe programs together to do what I want without
having to write a new one. I guess that's a fair reason to not compile :)
And a funny thing:
Today in the era of multi-core computers, not only the shell-only scripts are
unreadable, but they also are slower than scripts with pipes and little-to-no
shell features.
I tend to write scripts with only pipes. No shell features. And I also avoid
programs with many options (find(1)) when I can split it with xargs(1) and others.
I remember some script I wrote for maintenance of the Linux man-pages, and I
received a suggestion that I could "simplify" the script considerably and make
it "faster" by reducing the number of pipes using some features of find(1) or
sh(1) (I don't remember well). Well, I tested, and my long list of piped
programs outperformed by a fair amount the suggested alternative.
I didn't care enough to find out the reason, but I thought it could be because
with the pipeline, I could run each small process in a different core, so they
could work all at the same time. With a single program invocation, you're
bottle-necked by that program, which is limited to a single core (normally).
Cheers,
Alex
segaloco via
TUHS writes:
> I think that's a good point that scripting problems may be
> a symptom of the nature of the tools being used in them.
I think that you're hinting at something
different.
To the best of my recollection, scripting
languages were originally
intended and used for the automation of repetitive personal tasks;
making it easier for users who found themselves typing the same
stuff over and over again.
Indeed! Somewhere along the line people forgot
how to use a compiler and began writing large systems in a variety
of roughly equivalent but incompatible interpreted languages. Can
one even boot linux without having several different incompatible
versions of Python installed today? So I don't think that it's the
nature of the tools; I think that it's people choosing the wrong
tools for the problems that they're trying to solve.
Jon
The forgotten compilers were typically used to supply glue
to paste major tools together. The nature of that glue---often
simple data reformatting--inspired tools like sed and awk.
Each use of a tool became a process that saved many minutes
of work that would in a scriptless world be guided by hand,
boringly and unreliably.
Yet glue processes typically did only microseconds of
"real" work. In the name of efficiency, the operations began
to be incorporated directly into the shell. The first
inklings of this can be seen in "echo" and various forms
of variable-substitution making their way into the v7
shell. The phenomenon proliferated into putting what were
typically canned sed one-liners (but not sed itself) into
the shell.
Lots of specializations crowded out universality. A side
effect was an explosion of knowledge required to write
or understand code. Such is the tragedy of "forgetting
compilers".
Doug
5:19 p.m.
New subject: A few comments on porting the Bourne shell
Alejandro Colomar writes:
Well, as time goes by, I'm also writing less and
less programs. But only
because I find that I can pipe programs together to do what I want without
having to write a new one. I guess that's a fair reason to not compile :)
Wow! Small programs that do one thing and do it well connected together to
do more complicated things? What an awesome idea :-)
Jon
5 Jan
5 Jan
1:22 p.m.
New subject: A few comments on porting the Bourne shell
Jon Steinhart <jon(a)fourwinds.com> writes:
Wow! Small programs that do one thing and do it well
connected together to
do more complicated things? What an awesome idea :-)
*grin*
Sometimes people call it "screen scraping", and seem to think it's wrong
to desire parseable output from commands. I like to point them at this:
https://www.youtube.com/watch?v=tc4ROCJYbm0&t=358
-tih
--
Most people who graduate with CS degrees don't understand the significance
of Lisp. Lisp is the most important idea in computer science. --Alan Kay
9:11 p.m.
New subject: A few comments on porting the Bourne shell
In the v8 shell, the "whatis" command's output is all parseable by the
shell to recreate the item being identified. For instance, "whatis HOME"
would print
HOME=/usr/rob
while "whatis cd"
would print
builtin cd
and so on. I failed to convince POSIX or other forces to adopt this, and to
this day remain puzzled why people couldn't see its value, even though of
course not having an editable typescript does limit the flexibility
somewhat.
Plan 9 happened soon after.
-rob
On Fri, Jan 6, 2023 at 12:22 AM Tom Ivar Helbekkmo via TUHS <tuhs(a)tuhs.org>
wrote:
Jon Steinhart <jon(a)fourwinds.com> writes:
Wow! Small programs that do one thing and do it
well connected together
to
do more complicated things? What an awesome idea
:-)
*grin*
Sometimes people call it "screen scraping", and seem to think it's wrong
to desire parseable output from commands. I like to point them at this:
https://www.youtube.com/watch?v=tc4ROCJYbm0&t=358
-tih
--
Most people who graduate with CS degrees don't understand the significance
of Lisp. Lisp is the most important idea in computer science. --Alan Kay
3 Jan
3 Jan
5:26 p.m.
New subject: A few comments on porting the Bourne shell
On Tue, Jan 3, 2023 at 10:09 AM Douglas McIlroy
<douglas.mcilroy(a)dartmouth.edu> wrote:
The forgotten compilers were typically used to supply
glue
to paste major tools together. The nature of that glue---often
simple data reformatting--inspired tools like sed and awk.
Each use of a tool became a process that saved many minutes
of work that would in a scriptless world be guided by hand,
boringly and unreliably.
This, I think, begs the question I raised earlier: what precisely is
"the shell"? The command interpreter by itself isn't that interesting
without a supporting toolset. And now we get into the crux or what it
means to be "portable": what is the minimum set of such tools that
allow us to write portable shell scripts?
Again I ask, am I allowed to use `nawk`? How about '#!' lines? xargs?
cut? paste?
Perhaps that's the benefit of experience: over time, one develops a
sense of what works across the broadest range of systems that one will
likely encounter. But as Warner pointed out, we can be thankful we're
not using Eunice.
A few years ago, I was having lunch with some folks from the Go team
and one of them remarked, "you shouldn't write a shell script that's
longer than about 10 lines. Once you do, it's time to rewrite it in a
real programming language." I was a bit taken aback, but they had a
point. I'll note that Go standardized on using bash everywhere.
Yet glue processes typically did only microseconds of
"real" work. In the name of efficiency, the operations began
to be incorporated directly into the shell. The first
inklings of this can be seen in "echo" and various forms
of variable-substitution making their way into the v7
shell. The phenomenon proliferated into putting what were
typically canned sed one-liners (but not sed itself) into
the shell.
Perhaps I've told this story, but 10 or 12 years ago at Google a
colleague was writing a script to find the location of some resource
or other. I no longer recall what, exactly, but he had written this
massively over-structured thing: shell functions that ran one or two
commands, heavily commented, and he was trying to `return` the exit
status from those commands as the result from the function. It wasn't
working because, well, that's just not how such things work. I sat
down, and rewrote it into about 10 lines; the salient part was that
there was a conditional that determined whether the system it was
running on had a 64-bit or 32-bit userland and ran a handful of
commands depending. I showed him the revised script and he was
uncomfortable; after a while he kind of gave me a look and said,
"yeah, I just don't think people conceptualize of problems that way
anymore." Meaning, it was far more natural for him and others to just
write a Python program instead of a simple script. To him, the script
wasn't simple, but the Python program was.
That has stuck with me for a long, long time. Anyway, we checked the
script in and it worked; it's probably still running, though I suspect
most, if not all, 32-bit systems are gone from Google so perhaps it
was long-ago recycled. I have no idea anymore, but it really gave me
pause: had everything I'd learned about Unix in the prior 20 years
become obsolete? Was I an anachronism amongst my colleagues? Were they
right, and I wrong about how to approach problems?
Lots of specializations crowded out universality. A
side
effect was an explosion of knowledge required to write
or understand code. Such is the tragedy of "forgetting
compilers".
Something I've noticed is that lots of people try to increase
complexity to solve problems, and it rarely occurs to them to
eliminate complexity to solve problems. Sometimes the reasons for this
are good; most of the time they are not.
- Dan C.
6:07 p.m.
New subject: A few comments on porting the Bourne shell
On Tue, 3 Jan 2023, Dan Cross wrote:
Something I've noticed is that lots of people try
to increase
complexity to solve problems, and it rarely occurs to them to
eliminate complexity to solve problems. Sometimes the reasons for this
are good; most of the time they are not.
- Dan C.
I think of the saying: "Perfection is not when there is nothing left to
add, but when there is nothing left to remove."
-uso.
8:19 p.m.
New subject: A few comments on porting the Bourne shell
Steve Nickolas wrote in
<alpine.DEB.2.21.2301031307001.9988(a)sd-119843.dedibox.fr>fr>:
|On Tue, 3 Jan 2023, Dan Cross wrote:
|> Something I've noticed is that lots of people try to increase
|> complexity to solve problems, and it rarely occurs to them to
|> eliminate complexity to solve problems. Sometimes the reasons for this
|> are good; most of the time they are not.
|>
|> - Dan C.
|
|I think of the saying: "Perfection is not when there is nothing left to
|add, but when there is nothing left to remove."
He (Exupéry) was then shot down.
I always seem to response this to that.
Hmm, openpgp(a)ietf.org (to which i have almost zero to add
technically shall someone think that, nor do i want) lastly
|"Perfection is achieved not when there is nothing more to add but when \
|there
|is nothing left to take away" - Antoine de Saint-Exupéry.
He was then shot down.
But yes, he then really went missing.
The topic ..
I do not miss times where suddenly a shell script breaks because
": > FILE" does not work (just recently 'realized from reading code
of Paul Eggert of GNU/IANA TZ, "hey, > FILE" is of course
sufficient!"), i fixed it via "printf '' > FILE" by then;
whatever
the reason. May it be bugs or (local) miscompilations, not
detected due to missing unit tests and a too small user base.
Portable? If i find /usr/xpg4/bin i quickly add it to $PATH for
the much better awk (but beware of documented double expansion
issues) and the much much better sh(1).
Some things just require that, noclobber I/O redirection (set -C)
for example. (mktemp(1) is still not part of the POSIX standard.)
Besides i miss(ed) the history; the author of bmake (and
verieexec) just last year asked me why i would use stty for
a purpose ("(<&1 >/dev/null stty -a) 2>/dev/null") instead of
simply using "[ -t 1 ]", and indeed, i found that as soon as BSD
4.1 and Research V7, but it surprised me.
Without an oversight of the history and the lack of many systems
to test, perl(1) was omnipresent and if only for OpenSSL and so
using it for almost anything seemed save.
To love is not to look at one another: it is to look,
together, in the same direction. Antoine de Saint-Exupéry.
A happy and healthy new Year 2023 is now overdue.
Even Giants have to die, but with holding hands it can wait a bit
longer, i hope.
I wish that from Germany to all of you, and deliberately beyond
NATO readership.
--steffen
|
|Der Kragenbaer, The moon bear,
|der holt sich munter he cheerfully and one by one
|einen nach dem anderen runter wa.ks himself off
|(By Robert Gernhardt)
11:03 p.m.
New subject: A few comments on porting the Bourne shell
I apologize for this too long message, but context in this case matters.
tl;dr: I agree with the Go folks: past about 10 lines, write a program. But
not everybody will agree, as I learned with u-root. Many people love shell
programming. I never have.
u-root is a set of Go programs deployed on a couple million data center
systems at many companies, including Google.
u-root originally used a compile-on-demand model: type a command name, if
it's not in /ubin, it gets built. This was fast, in early Go, typically 250
ms or so. And, in the early days of Go, the Go toolchain, u-root, and all
support source easily fit in a 16M SPI part.
The original scripting language for u-root was Go.
There were two commands for this: script and builtin. script would run Go
fragments; builtin would take supplied commands and build a custom shell
with those built in. Each took 250ms to run.
script took what we called a 'Go fragment', wrapped it with boiler plate,
compiled it, and ran it.
e.g.
script fmt.Printf("hi\n")
would build the program and run that code.
So you could, e.g,, do math:
script fmt.Printf("%d\n", 6*7)
It could get complex: to see things about interfaces:
script 'ifaces, _ := net.Interfaces()
for _, v := range ifaces {
addrs, _ := v.Addrs()
Printf("%v has %v", v, addrs)
}'
you'd get:
ip: {1 1500 lo up|loopback} has [127.0.0.1/8 ::1/128]
ip: {5 1500 eth0 fa:42:2c:d4:0e:01 up|broadcast} has [172.17.0.2/16
fe80::f842:2cff:fed4:e01/64]
The second command was called builtin. It did not work as other shells do:
it built a new shell for you. So, you type:
bulitin hi fmt.Printf("hi") there fmt.Printf("there\n")
builtin would convert the Go fragments to functions callable in the u-root
shell, build a private name space (on Linux or Plan 9 anyway), rebuild the
shell with those new functions, and at that point:
you type
hi
in the shell, and it types
hi
back. This was built in to your private shell in your private name space.
Once you left the shell, it was gone.
Again, this process of creating and starting the new shell always took
about 250 ms (in Go 1.2 that is).
I learned a lesson: people love their shell scripting languages. Nobody
wanted to script with Go. It made me sad, but that's how it Go-es.
ron
p.s. the 'script' command is still available as an experimental u-root
command. Source mode is now independent: github.com/u-root/sourcery
On Tue, Jan 3, 2023 at 2:35 PM Steffen Nurpmeso <steffen(a)sdaoden.eu> wrote:
Steve Nickolas wrote in
<alpine.DEB.2.21.2301031307001.9988(a)sd-119843.dedibox.fr>fr>:
|On Tue, 3 Jan 2023, Dan Cross wrote:
|> Something I've noticed is that lots of people try to increase
|> complexity to solve problems, and it rarely occurs to them to
|> eliminate complexity to solve problems. Sometimes the reasons for this
|> are good; most of the time they are not.
|>
|> - Dan C.
|
|I think of the saying: "Perfection is not when there is nothing left to
|add, but when there is nothing left to remove."
He (Exupéry) was then shot down.
I always seem to response this to that.
Hmm, openpgp(a)ietf.org (to which i have almost zero to add
technically shall someone think that, nor do i want) lastly
|"Perfection is achieved not when there is nothing more to add but when
\
|there
|is nothing left to take away" - Antoine de Saint-Exupéry.
He was then shot down.
But yes, he then really went missing.
The topic ..
I do not miss times where suddenly a shell script breaks because
": > FILE" does not work (just recently 'realized from reading code
of Paul Eggert of GNU/IANA TZ, "hey, > FILE" is of course
sufficient!"), i fixed it via "printf '' > FILE" by then;
whatever
the reason. May it be bugs or (local) miscompilations, not
detected due to missing unit tests and a too small user base.
Portable? If i find /usr/xpg4/bin i quickly add it to $PATH for
the much better awk (but beware of documented double expansion
issues) and the much much better sh(1).
Some things just require that, noclobber I/O redirection (set -C)
for example. (mktemp(1) is still not part of the POSIX standard.)
Besides i miss(ed) the history; the author of bmake (and
verieexec) just last year asked me why i would use stty for
a purpose ("(<&1 >/dev/null stty -a) 2>/dev/null") instead of
simply using "[ -t 1 ]", and indeed, i found that as soon as BSD
4.1 and Research V7, but it surprised me.
Without an oversight of the history and the lack of many systems
to test, perl(1) was omnipresent and if only for OpenSSL and so
using it for almost anything seemed save.
To love is not to look at one another: it is to look,
together, in the same direction. Antoine de Saint-Exupéry.
A happy and healthy new Year 2023 is now overdue.
Even Giants have to die, but with holding hands it can wait a bit
longer, i hope.
I wish that from Germany to all of you, and deliberately beyond
NATO readership.
--steffen
|
|Der Kragenbaer, The moon bear,
|der holt sich munter he cheerfully and one by one
|einen nach dem anderen runter wa.ks himself off
|(By Robert Gernhardt)
4 Jan
4 Jan
1:37 a.m.
New subject: A few comments on porting the Bourne shell
I write shell one liners all the time[1].
Go is just too verbose for scripting. I use it for larger programs but typically these
don't start out as shell scripts.
[1] In fact I use find (or zsh's **/*), xargs, grep, sort, uniq, field picking often
enough that I wish they were built-in a shell but I haven't figured out a simple
syntax for it as yet.
On Jan 3, 2023, at 3:03 PM, ron minnich
<rminnich(a)gmail.com> wrote:
I apologize for this too long message, but context in this case matters.
tl;dr: I agree with the Go folks: past about 10 lines, write a program. But not everybody
will agree, as I learned with u-root. Many people love shell programming. I never have.
u-root is a set of Go programs deployed on a couple million data center systems at many
companies, including Google.
u-root originally used a compile-on-demand model: type a command name, if it's not
in /ubin, it gets built. This was fast, in early Go, typically 250 ms or so. And, in the
early days of Go, the Go toolchain, u-root, and all support source easily fit in a 16M SPI
part.
The original scripting language for u-root was Go.
There were two commands for this: script and builtin. script would run Go fragments;
builtin would take supplied commands and build a custom shell with those built in. Each
took 250ms to run.
script took what we called a 'Go fragment', wrapped it with boiler plate,
compiled it, and ran it.
e.g.
script fmt.Printf("hi\n")
would build the program and run that code.
So you could, e.g,, do math:
script fmt.Printf("%d\n", 6*7)
It could get complex: to see things about interfaces:
script 'ifaces, _ := net.Interfaces()
for _, v := range ifaces {
addrs, _ := v.Addrs()
Printf("%v has %v", v, addrs)
}'
you'd get:
ip: {1 1500 lo up|loopback} has [127.0.0.1/8 <http://127.0.0.1/8> ::1/128]
ip: {5 1500 eth0 fa:42:2c:d4:0e:01 up|broadcast} has [172.17.0.2/16
<http://172.17.0.2/16> fe80::f842:2cff:fed4:e01/64]
The second command was called builtin. It did not work as other shells do: it built a new
shell for you. So, you type:
bulitin hi fmt.Printf("hi") there fmt.Printf("there\n")
builtin would convert the Go fragments to functions callable in the u-root shell, build a
private name space (on Linux or Plan 9 anyway), rebuild the shell with those new
functions, and at that point:
you type
hi
in the shell, and it types
hi
back. This was built in to your private shell in your private name space. Once you left
the shell, it was gone.
Again, this process of creating and starting the new shell always took about 250 ms (in
Go 1.2 that is).
I learned a lesson: people love their shell scripting languages. Nobody wanted to script
with Go. It made me sad, but that's how it Go-es.
ron
p.s. the 'script' command is still available as an experimental u-root command.
Source mode is now independent: github.com/u-root/sourcery
<http://github.com/u-root/sourcery>
On Tue, Jan 3, 2023 at 2:35 PM Steffen Nurpmeso <steffen(a)sdaoden.eu
<mailto:steffen@sdaoden.eu>> wrote:
> Steve Nickolas wrote in
> <alpine.DEB.2.21.2301031307001.9988(a)sd-119843.dedibox.fr
<mailto:alpine.DEB.2.21.2301031307001.9988@sd-119843.dedibox.fr>>:
> |On Tue, 3 Jan 2023, Dan Cross wrote:
> |> Something I've noticed is that lots of people try to increase
> |> complexity to solve problems, and it rarely occurs to them to
> |> eliminate complexity to solve problems. Sometimes the reasons for this
> |> are good; most of the time they are not.
> |>
> |> - Dan C.
> |
> |I think of the saying: "Perfection is not when there is nothing left to
> |add, but when there is nothing left to remove."
>
> He (Exupéry) was then shot down.
>
> I always seem to response this to that.
> Hmm, openpgp(a)ietf.org <mailto:openpgp@ietf.org> (to which i have almost zero to
add
> technically shall someone think that, nor do i want) lastly
>
> |"Perfection is achieved not when there is nothing more to add but when \
> |there
> |is nothing left to take away" - Antoine de Saint-Exupéry.
>
> He was then shot down.
>
> But yes, he then really went missing.
>
> The topic ..
> I do not miss times where suddenly a shell script breaks because
> ": > FILE" does not work (just recently 'realized from reading
code
> of Paul Eggert of GNU/IANA TZ, "hey, > FILE" is of course
> sufficient!"), i fixed it via "printf '' > FILE" by then;
whatever
> the reason. May it be bugs or (local) miscompilations, not
> detected due to missing unit tests and a too small user base.
>
> Portable? If i find /usr/xpg4/bin i quickly add it to $PATH for
> the much better awk (but beware of documented double expansion
> issues) and the much much better sh(1).
> Some things just require that, noclobber I/O redirection (set -C)
> for example. (mktemp(1) is still not part of the POSIX standard.)
>
> Besides i miss(ed) the history; the author of bmake (and
> verieexec) just last year asked me why i would use stty for
> a purpose ("(<&1 >/dev/null stty -a) 2>/dev/null") instead of
> simply using "[ -t 1 ]", and indeed, i found that as soon as BSD
> 4.1 and Research V7, but it surprised me.
> Without an oversight of the history and the lack of many systems
> to test, perl(1) was omnipresent and if only for OpenSSL and so
> using it for almost anything seemed save.
>
> To love is not to look at one another: it is to look,
> together, in the same direction. Antoine de Saint-Exupéry.
>
> A happy and healthy new Year 2023 is now overdue.
> Even Giants have to die, but with holding hands it can wait a bit
> longer, i hope.
> I wish that from Germany to all of you, and deliberately beyond
> NATO readership.
>
> --steffen
> |
> |Der Kragenbaer, The moon bear,
> |der holt sich munter he cheerfully and one by one
> |einen nach dem anderen runter wa.ks himself off
> |(By Robert Gernhardt)
1:58 a.m.
New subject: A few comments on porting the Bourne shell
On Jan 3, 2023, at 6:37 PM, Bakul Shah
<bakul(a)iitbombay.org> wrote:
I write shell one liners all the time[1].
Go is just too verbose for scripting. I use it for larger programs but typically these
don't start out as shell scripts.
[1] In fact I use find (or zsh's **/*), xargs, grep, sort, uniq, field picking
often enough that I wish they were built-in a shell but I haven't figured out a
simple syntax for it as yet.
I was 50 years old before I learned that sort has a -u, so I could have saved the
pipe-through-uniq step.
I also keep getting told grep has a -r but my finger macros won't stop typing find .
-name \*.py | xargs ...
(And yes, it's GNU find just about everywhere, so I can do -print0 | xargs -0 and not
have to get fancy with the -I)
But also, crap, it was 25, count 'em, 25 years ago that a friend looked at my
workflow and said "I've never seen anyone use a GUI just as a way to keep a
dozen terminal windows open side by side before."
Adam
3:19 p.m.
New subject: A few comments on porting the Bourne shell
Hi Adam,
Bakul Shah wrote:
> In fact I use find (or zsh's **/*)
bash(1) also has ‘**’ if ‘shopt -s globstar’.
I was 50 years old before I learned that sort has a
-u, so I could
have saved the pipe-through-uniq step.
Did you never have a Unix with fine man pages to read? :-) Before info(1).
Or ‘The Unix Programming Environment’ mentions sort(1)'s -u on page 106.
I also keep getting told grep has a -r but my finger
macros won't stop
typing find . -name \*.py | xargs ...
‘... **/*.py’ using the ‘**’ Bakul mentions above.
(And yes, it's GNU find just about everywhere, so
I can do -print0 |
xargs -0 and not have to get fancy with the -I)
A lot of the time, POSIX find's ‘-exec foo {} +’ suffices and runs
foo with as many arguments as will just fit under argv[]s limits,
like xargs by default and unlike find's one-at-a-time behaviour with
‘-exec foo {} \;’.
"I've never seen anyone use a GUI just as a
way to keep a
dozen terminal windows open side by side before."
Clearly, you're a trendsetter. :-) Tiling window managers are popular,
e.g. https://en.wikipedia.org/wiki/I3_(window_manager)
--
Cheers, Ralph.
6:01 p.m.
New subject: A few comments on porting the Bourne shell
On Jan 4, 2023, at 7:19 AM, Ralph Corderoy <ralph(a)inputplus.co.uk> wrote:
A lot of the time, POSIX find's ‘-exec foo {} +’ suffices and runs
foo with as many arguments as will just fit under argv[]s limits,
like xargs by default and unlike find's one-at-a-time behaviour with
‘-exec foo {} \;’.
I often run further transformations before executing
some command on selected files. For example
find . -type f -name '*.[csh]' | grep -l foo | xargs wc -l
Composability rules!
8:46 p.m.
New subject: A few comments on porting the Bourne shell
Hello Bakul, Ralph,
On 1/4/23 19:01, Bakul Shah wrote:
On Jan 4, 2023, at 7:19 AM, Ralph Corderoy
<ralph(a)inputplus.co.uk> wrote:
I find find(1)'s options too complex. I prefer composing even more. I very
often find myself writing:
find . -type f | grep '\.[ch]$' | grep -l foo | xargs ...
A lot of the time, POSIX find's ‘-exec foo {} +’ suffices and runs
foo with as many arguments as will just fit under argv[]s limits,
like xargs by default and unlike find's one-at-a-time behaviour with
‘-exec foo {} \;’.
I often run further transformations before executing
some command on selected files. For example
find . -type f -name '*.[csh]' | grep -l foo | xargs wc -l
Composability rules!
Yup it does :)
For me, the only shell feature that matters is the pipe. I also like using
bash(1) for it's pipefail feature (and I also usually enable lastpipe). But
things like globbing and regex support, I don't like them at all.
I wish shells didn't supoprt globbing, and that glob(1) would be a standalone
program still today. It would simplify much of the quoting issues if most
characters were just characters to the shell.
Cheers,
Alex
5 Jan
5 Jan
12:06 a.m.
New subject: A few comments on porting the Bourne shell
On Wed, Jan 4, 2023 at 3:47 PM Alejandro Colomar <alx.manpages(a)gmail.com>
wrote:
I wish shells didn't supoprt globbing, and that glob(1) would be a
standalone
program still today. It would simplify much of the quoting issues if most
characters were just characters to the shell.
I like the way the rc shell works: globbing is done, but any word in single
quotes suppresses all interpretation (double quote has no meaning)
12:41 a.m.
New subject: A few comments on porting the Bourne shell
I was today years old when I learned that GNU sort has a -h option. Which
doesn't mean "help" but means "use human-friendly units."
Except I sort of derived its existence from first principles. Someone
wanted a way to find disk hogs (they were on a Linux box), but also wanted
SI units, and I said to myself, "self, doesn't that sound like exactly the
sort of baroque and vaguely nauseating option that GNU would add to sort?"
Turns out it is.
Well, I mean, the MacOS man page just says that -h is an extension to POSIX
(yes, it's there on MacOS too), but I will eat a small hat of mine if it
didn't come to sort via GNU.
Adam
On Wed, Jan 4, 2023 at 5:07 PM John Cowan <cowan(a)ccil.org> wrote:
On Wed, Jan 4, 2023 at 3:47 PM Alejandro Colomar <alx.manpages(a)gmail.com>
wrote:
I wish shells didn't supoprt globbing, and that glob(1) would be a
standalone
program still today. It would simplify much of the quoting issues if
most
characters were just characters to the shell.
I like the way the rc shell works: globbing is done, but any word in
single quotes suppresses all interpretation (double quote has no meaning)
4 Jan
4 Jan
5:05 a.m.
New subject: A few comments on porting the Bourne shell
On Tue, Jan 03, 2023 at 03:03:15PM -0800, ron minnich wrote:
I learned a lesson: people love their shell scripting
languages. Nobody
wanted to script with Go. It made me sad, but that's how it Go-es.
Speaking as someone who has written and maintaining thousands of lines
of bash shell script to support file system testing[1] for Linux,
there's a really simple reason for it. In order to manipulate GCE
VM's, it's just **way** easier to do it from a shell.
[1] https://thunk.org/gce-xfstests
Compare:
gcloud compute instances create VM_NAME \
[--image=IMAGE | --image-family=IMAGE_FAMILY] \
--image-project=IMAGE_PROJECT
--machine-type=MACHINE_TYPE
with the equivalent in in other "real languages", per [2]. It's 53
lines of C++, 64 lines of Go code, or 100 lines in Java, or 242(!)
lines in Python.
[2] https://cloud.google.com/compute/docs/instances/create-start-instance#expan…
Sure, it might a few thousands lines of codes of bash script. But it
would probably be tens of thousands of lines in Go or Python.
Furthermore, when I'm trying to experiment with how to experiment with
various GCE SDK interfaces, using gcloud and shell is way, way faster
than trying to experiment with writing go code, multiple rounds of
trying to compile the go code, fixing compile syntax errors, etc ---
versus typing a few hundreds characters of a single gcloud command
line, and adjusting CLI options untll it does what you want.
For my use case, I **really** only care about it running on Linux,
since it's being used to test Linux file system kernel code. I have a
job I'm trying to get done, which is to write reliable ext4 file
system code. Implementing kvm-xfstests[2], gce-xfstests[3], and
android-xfstests[4] are just a means to an end.
[2] https://github.com/tytso/xfstests-bld/blob/master/Documentation/kvm-quickst…
[3] https://github.com/tytso/xfstests-bld/blob/master/Documentation/gce-xfstest…
[4] https://github.com/tytso/xfstests-bld/blob/master/Documentation/android-xfs…
My time is valuable, and I'm not getting paid to write a general file
system test framework that could potentially work on FreeBSD or
NetBSD. If my developer velocity is faster by implementing it in
bash, that's what I'm going to do, since time spent rewriting
gce-xfstests in go or Python is time that I can't spend doing work on
the ext4 file system.
If someone wants to try to reimplementing my bash scripts in go, they
should feel free to send a patch. :-)
- Ted
3 Jan
3 Jan
6:19 p.m.
New subject: A few comments on porting the Bourne shell
Den tis 3 jan. 2023 kl 18:28 skrev Dan Cross <crossd(a)gmail.com>om>:
A few years ago, I was having lunch with some folks
from the Go team
and one of them remarked, "you shouldn't write a shell script that's
longer than about 10 lines. Once you do, it's time to rewrite it in a
real programming language." I was a bit taken aback, but they had a
point. I'll note that Go standardized on using bash everywhere.
They wouldn't like my current project. It's a whole bunch of different
shell scripts (and they use bash-isms, since it's inconceivable that anyone
would use them on a non-Linux non-GNU system) calling each other as
modules. The scripts aren't HUGE, but they're certainly a lot more than 10
lines.
I work for an academic supercomputing facility, and we have a corporate
customer using our systems to do aerodynamics calculations. They want a way
to automatically submit jobs on our systems from their own environment,
complete with transferring the source data files to us and the results
files back to them when they're done. They're integrating that with their
own set of scripts (a lot of Python, I believe) so an aerodynamics engineer
can just pick an option from a menu and everything works automagically. I'd
say bash works pretty well for that application. It's a lot of calling
external programs (scp, sbatch (the command to submit a SLURM job, SLURM
being the batch management facility a lot of large compute clusters use to
manage their jobs), etc).
I inherited this set of scripts from someone who quit before I joined in
2020, and have modified them extensively since then to work well with the
corporate customer's scripts and be more generic. It's in a fully working
state at this point, but I still have some feature requests to take care of.
I am more of a sysadmin than a developer, but I certainly can write code
when the situation warrants. Pure Windows sysadmins often can't write code
at all (though there are certainly some who are very conversant with
PowerShell) but I'd say any UNIX/Linux sysadmin worth their salt can at
least write shell scripts, and often Python, or some years back, Perl.
Niklas
4 Jan
4 Jan
1:29 a.m.
New subject: A few comments on porting the Bourne shell
On Jan 3, 2023, at 10:26 AM, Dan Cross
<crossd(a)gmail.com> wrote:
A few years ago, I was having lunch with some folks from the Go team
and one of them remarked, "you shouldn't write a shell script that's
longer than about 10 lines. Once you do, it's time to rewrite it in a
real programming language." I was a bit taken aback, but they had a
point. I'll note that Go standardized on using bash everywhere.
My number is larger than 10, but it's smaller than 100.
Lord knows I have plenty of things that started as a pipe and-or-for-loop at the command
line, that became a shell script, that grew some arguments, that got beefier, and that I
then rewrote in Perl or Python and cursed myself for not having done it that way from the
start.
The RSP Notebook I mentioned on COFF is about to have its installer--which is a hodgepodge
of shell scripts and Python scripts--rewritten as reasonable Python classes with typing
and test cases and everything, because it's grown enough that it's hard to
understand in its current form.
Adam
5 Jan
5 Jan
1:51 a.m.
New subject: A few comments on porting the Bourne shell
On 1/4/23 02:29, Adam Thornton wrote:
> On Jan 3, 2023, at 10:26 AM, Dan Cross
<crossd(a)gmail.com> wrote:
>
>
> A few years ago, I was having lunch with some folks from the Go team
> and one of them remarked, "you shouldn't write a shell script that's
> longer than about 10 lines. Once you do, it's time to rewrite it in a
> real programming language." I was a bit taken aback, but they had a
> point. I'll note that Go standardized on using bash everywhere.
I have many counterexamples that take more than 10 lines. I'll show a couple
that are very useful to me:
# man_section() prints specific manual page sections (DESCRIPTION, SYNOPSIS,
# ...) of all manual pages in a directory (or in a single manual page file).
# Usage example: .../man-pages$ man_section man2 SYNOPSIS 'SEE ALSO';
man_section()
{
if [ $# -lt 2 ]; then
&2 echo "Usage: ${FUNCNAME[0]} <dir>
<section>...";
return $EX_USAGE;
fi
local page="$1";
shift;
local sect="$*";
find "$page" -type f \
|xargs wc -l \
|grep -v -e '\b1 ' -e '\btotal\b' \
|awk '{ print $2 }' \
|sort \
|while read -r manpage; do
(sed -n '/^\.TH/,/^\.SH/{/^\.SH/!p}' <"$manpage";
for s in $sect; do
<"$manpage" \
sed -n \
-e "/^\.SH $s/p" \
-e "/^\.SH $s/,/^\.SH/{/^\.SH/!p}";
done;) \
|man -P cat -l - 2>/dev/null;
done;
}
I don't think it's unreadable by being too long, and in every other language it
would take a lot more work to implement. And here's another one, which I used a
lot until I wrote grepc(1)[1]:
# grep_syscall() finds the prototype of a syscall in the kernel sources,
# printing the filename, line number, and the prototype.
# It should be run from the root of the linux kernel source tree.
# Usage example: .../linux$ grep_syscall openat2;
#
# See also: grepc(1)
grep_syscall()
{
if [ $# -ne 1 ]; then
&2 echo "Usage: ${FUNCNAME[0]}
<syscall>";
return $EX_USAGE;
fi
find ./* -type f \
|grep '\.c$' \
|xargs grep -l "$1" \
|sort \
|xargs pcregrep -Mn "(?s)^\w*SYSCALL_DEFINE.\($1\b.*?\)" \
|sed -E 's/^[^:]+:[0-9]+:/&\n/';
find ./* -type f \
|grep '\.[ch]$' \
|xargs grep -l "$1" \
|sort \
|xargs pcregrep -Mn "(?s)^asmlinkage\s+[\w\s]+\**sys_$1\s*\(.*?\)" \
|sed -E 's/^[^:]+:[0-9]+:/&\n/';
}
Cheers,
Alex
[1]: <http://www.alejandro-colomar.es/src/alx/alx/grepc.git/>
My number is larger than 10, but it's smaller than 100.
474
days inactive
476
days old
19 comments
15 participants
participants (15)
-
Adam Thornton -
Alejandro Colomar -
Bakul Shah -
Dan Cross -
Douglas McIlroy -
John Cowan -
Jon Steinhart -
Niklas Karlsson -
Ralph Corderoy -
Rob Pike -
ron minnich -
Steffen Nurpmeso -
Steve Nickolas -
Theodore Ts'o -
Tom Ivar Helbekkmo