TUHS

tuhs@tuhs.org

40 participants
6499 discussions

Systematic approach to command-line interfaces

by Michael Siegel

Hello, I've recently started to implement a set of helper functions and procedures for parsing Unix-like command-line interfaces (i.e., POSIX + GNU-style long options, in this case) in Ada. While doing that, I learned that there is a better way to approach this problem – beyond using getopt(s) (which never really made sense to me) and having to write case statements in loops every time: Define a grammar, let a pre-built parser do the work, and have the parser provide the results to the program. Now, defining such a grammar requires a thoroughly systematic approach to the design of command-line interfaces. One problem with that is whether that grammar should allow for sub-commands. And that leads to the question of how task-specific tool sets should be designed. These seem to be a relatively new phenomenon in Unix-like systems that POSIX doesn't say anything about, as far as I can see. So, I've prepared a bit of a write-up, pondering on the pros and cons of two different ways of having task-specific tool sets (non-hierarchical command sets vs. sub-commands) that is available at https://www.msiism.org/files/doc/unix-like_command-line_interfaces.html I tend to think the sub-command approach is better. But I'm neither a UI nor a Unix expert and have no formal training in computer things. So, I thought this would be a good place to ask for comment (and get some historical perspective). This is all just my pro-hobbyist attempt to make some people's lives easier, especially mine. I mean, currently, the "Unix" command line is quite a zoo, and not in a positive sense. Also, the number of well-thought-out command-line interfaces doesn't seem to be a growing one. But I guess that could be changed by providing truly easy ways to make good interfaces. -- Michael

3 years, 7 months

Re: [TUHS] Systematic approach to command-line interfaces

by Douglas McIlroy

> one other thing that SLS breaks, for data files, is the whole Unix 'pipe' > abstraction, which is at the heart of the whole Unix tools paradigm. Multics had an IO system with an inherent notion of redirectable data streams. Pipes could have--and eventually did (circa 1987)--fit into that framework. I presume a pipe DIM (device interface manager) was not hard to build once it was proposed and accepted. Doug

3 years, 7 months

Re: [TUHS] Systematic approach to command-line interfaces

by jnc＠mercury.lcs.mit.edu

> From: Larry McVoy > If you read(2) a page and mmap()ed it and then did a write(2) to the > page, the mapped page is the same physical memory as the write()ed > page. Zero coherency issues. Now I'm confused; read() and write() semantically include a copy operation (so there are then two copies of that data chunk, and possible consistency issues between them), and the copied item is not necessarily page-sized (so you can't ensure consistency between the original+copy by mapping it in). So when one does a read(file, &buffer, 1), one gets a _copy of just that byte_ in the process' address space (and similar for write()). Yes, there's no coherency issue between the contents of an mmap()'d page, and the system's idea of what's in that page of the file, but that's a _different_ coherency issue. Or am I confused? PS: > From: "Greg A. Woods" > I now struggle with liking the the Unix concept of "everything is a > file" -- especially with respect to actual data files. Multics also got > it right to use single-level storage -- that's the right abstraction Oh, one other thing that SLS breaks, for data files, is the whole Unix 'pipe' abstraction, which is at the heart of the whole Unix tools paradigm. So no more 'cmd | wc' et al. And since SLS doesn't have the 'make a copy' semantics of pipe output, it would be hard to trivially work around it. Yes, one could build up a similar framework, but each command would have to specify an input file and an output file (no more 'standard in' and 'out'), and then the command interpreter would have to i) take command A's output file and feed it to command B, and ii) delete A's output file when the whole works was done. Yes, the user could do it manually, but compare: cmd aaa | wc and cmd aaa bbb wc bbb rm bbb If bbb is huge, one might run out of room, but with today's 'light my cigar with disk blocks' life, not a problem - but it would involve more disk traffic, as bbb would have to be written out in its entirety, not just have a mall piece kept in the disk cache as with a pipe. Noel

3 years, 7 months

Re: [TUHS] Systematic approach to command-line interfaces

by jnc＠mercury.lcs.mit.edu

> From: "Greg A. Woods" > the elegance of fork() is incredible! That's because in PDP-11 Unix, they didn't have the _room_ to create a huge mess. Try reading the exec() code in V6 or so. (I'm in a bit of a foul mood today; my laptop sorta locked up when a _single_ Edge browser window process grew to almost _2GB_ in size. Are you effing kidding me? If I had any idea what today would look like, back when I was 20 - especially the massive excrement pile that the Internet has turned into - I never would have gone into computers - cabinetwork, or something, would have been an infinitely superior career choice.) > I now struggle with liking the the Unix concept of "everything is a > file" -- especially with respect to actual data files. Multics also got > it right to use single-level storage -- that's the right abstraction Well, files a la Unix, instead of the SLS, are OK for a _lot_ of data storage - pretty much everything except less-common cases like concurrent access to a shared database, etc. Where the SLS really shines is _code_ - being able to just do a subroutine call to interact with something else has incredible bang/buck ratio - although I concede doing it all securely is hard (although they did make a lot of progress there). Noel

3 years, 7 months

Thompson trojan put into practice

by Douglas McIlroy

>> > It's part of my academic project to work on provable compiler security. >> > I tried to do it according to the "Reflections on Trusting Trust" by Ken >> > Thompson, not only to show a compiler Trojan horse but also to prove that >> > we can discover it. >> >> Of course it can be discovered if you look for it. What was impressive about >> the folks who got Thompson's compiler at PWB is that they found the horse >> even though they weren't looking for it. > I had not heard this story. Can you elaborate, please? My impression from having > read the paper (a long time ago now) is that Ken did the experiment locally only. Ken did it locally, but a vigilant person at PWB noticed there was an experimental compiler on the research machine and grabbed it. While they weren't looking for hidden stuff, they probably were trying to find what was new in the compiler. Ken may know details about what they had in the way of source and binary. Doug

3 years, 8 months

Thompson trojan put into practice

by Douglas McIlroy

> It's part of my academic project to work on provable compiler security. > I tried to do it according to the "Reflections on Trusting Trust" by Ken > Thompson, not only to show a compiler Trojan horse but also to prove that > we can discover it. Of course it can be discovered if you look for it. What was impressive about the folks who got Thompson's compiler at PWB is that they found the horse even though they weren't looking for it. Then there was the first time Jim Reeds and I turned on integrity control in IX, our multilevel-security version of Research Unix. When it reported a security violation during startup we were sure it was a bug. But no, it had snagged Tom Duff's virus in the act of replication. It surprised Tom as much as it did us, because he thought he'd eradicated it. Doug

3 years, 8 months

Re: [TUHS] Thompson trojan put into practice

by jnc＠mercury.lcs.mit.edu

https://googlethatforyou.com/?q=Tom%20Duff%20Virus Noel

3 years, 8 months

Thompson trojan put into practice

by arnold＠skeeve.com

This is FYI. No comment on whether it was a good idea or not. :-) Arnold > From: Niklas Rosencrantz <niklasro(a)gmail.com> > Date: Sun, 19 Sep 2021 17:10:24 +0200 > To: tinycc-devel(a)nongnu.org > Subject: Re: [Tinycc-devel] Can tcc compile itself with Apple M1? > > > Hello! > > For demonstration purpose I put my experiment with a compiler backdoor in a > public repository > https://github.com/montao/ddc-tinyc/blob/857d927363e9c9aaa713bb20adbe99ded7… > > It's part of my academic project to work on provable compiler security. > I tried to do it according to the "Reflections on Trusting Trust" by Ken > Thompson, not only to show a compiler Trojan horse but also to prove that > we can discover it. > What it does is inject arbitrary code to the next version of the compiler > and so on. > > Regards \n

3 years, 8 months

ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe)

by Dan Cross

One of the things I really appreciate about participating in this community and studying Unix history (and the history of other systems) is that it gives one firm intellectual ground from which to evaluate where one is going: without understanding where one is and where one has been, it's difficult to assert that one isn't going sideways or completely backwards. Maybe either of those outcomes is appropriate at times (paradigms shift; we make mistakes; etc) but generally we want to be moving mostly forward. The danger when immersing ourselves in history, where we must consider and appreciate the set of problems that created the evolutionary paths leading to the systems we are studying, is that our thinking can become calcified in assuming that those systems continue to meet the needs of the problems of today. It is therefore always important to reevaluate our base assumptions in light of either disconfirming evidence or (in our specific case) changing environments. To that end, I found Timothy Roscoe's (ETH) joint keynote address at ATC/OSDI'21 particularly compelling. He argues that what we consider the "operating system" is only controlling a fraction of a modern computer these days, and that in many ways our models for what we consider "the computer" are outdated and incomplete, resulting in systems that are artificially constrained, insecure, and with separate components that do not consider each other and therefore frequently conflict. Further, hardware is ossifying around the need to present a system interface that can be controlled by something like Linux (used as a proxy more generally for a Unix-like operating system), simultaneously broadening the divide and making it ever more entrenched. Another theme in the presentation is that, to the limited extent the broader systems research community is actually approaching OS topics at all, it is focusing almost exclusively on Linux in lieu of new, novel systems; where non-Linux systems are featured (something like 3 accepted papers between SOSP and OSDI in the last two years out of $n$), the described systems are largely Linux-like. Here the presentation reminded me of Rob Pike's "Systems Software Research is Irrelevant" talk (slides of which are available in various places, though I know of no recording of that talk). Roscoe's challenge is that all of this should be seen as both a challenge and an opportunity for new research into operating systems specifically: what would it look like to take a holistic approach towards the hardware when architecting a new system to drive all this hardware? We have new tools that can make this tractable, so why don't we do it? Part of it is bias, but part of it is that we've lost sight of the larger picture. My own question is, have we become entrenched in the world of systems that are "good enough"? Things he does NOT mention are system interfaces to userspace software; he doesn't seem to have any quibbles with, say, the Linux system call interface, the process model, etc. He's mostly talking about taking into account the hardware. Also, in fairness, his highlighting a "small" portion of the system and saying, "that's what the OS drives!" sort of reminds me of the US voter maps that show vast tracts of largely unpopulated land colored a certain shade as having voted for a particular candidate, without normalizing for population (land doesn't vote, people do, though in the US there is a relationship between how these things impact the overall election for, say, the presidency). I'm curious about other peoples' thoughts on the talk and the overall topic? https://www.youtube.com/watch?v=36myc8wQhLo - Dan C.

3 years, 8 months

RegExp decision for meta characters: Circumflex

by Douglas McIlroy

> Maybe there existed RE notations that were simply copied ... Ed was derived from Ken's earlier qed. Qed's descendant in Multics was described in a 1969 GE document: http://www.bitsavers.org/pdf/honeywell/multics/swenson/6906.multics-condens…. Unfortunately it describes regular expressions only sketchily by example. However, alternation, symbolized by | with grouping by parentheses, was supported in qed, whereas alternation was omitted from ed. The GE document does not mention character classes; an example shows how to use alternation for the same purpose. Beginning-of-line is specified by a logical-negation symbol. In apparent contradiction, the v1 manual says the meanings of [ and ^ are the same in ed and (an unspecified version of) qed. My guess about the discrepancies is no better than yours. (I am amused by the title "condensed guide" for a manual in which each qed request gets a full page of explanation. It exemplifies how Unix split from Multics in matters of taste.) Doug

3 years, 8 months

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

1996

1995

1994

1993

1992

1991

1990

TUHS