On 2017-05-05 10:16 PM, Noel Hunt wrote: Much as I love Professor Knuth, he is still doing this. He always was a low level coder*, and his programs are longer and more complicated than the high level equivalents, by the expected factors. (If anyone doubts it, note that he adopted C after Pascal, and many of his programs are linked from his web pages.) No doubt he has solid rationalisations for taking this approach for his purposes, but of course it's not ideal for all situations. --Toby * - Many of us will remember we were expected to understand TAOCP through MIX. And he started out like this: http://www.catonmat.net/blog/donald-knuths-first-computer/

 few more comments about style and optimization:First, the thing that make optimized code hard to understand is that (at least when a compiler does it) the optimized program goes through a different set of states than the user expected from the source code.  When a loop is unrolled, or invariant expressions moved out of a loop, or indexing is turned into pointer manipulation, if you don't know what's happening its mysterious.  But once you'v seen this a few times, it's only a momentary hitch...Also, one lesson I learned from Unix is that if you can make a program 10x faster it becomes a qualitatively different experience.  The first version of Yacc I wrote, using the textbook algorithms, took 20 minutes to process a 30-rule grammar.  On the shared PDP-11, when I ran Yacc everyone else's jobs slowed to a crawl, and people were heard to mutter "*%&#!  Johnson's running Yacc again!"   A couple of years later, Yacc produced parsers faster than the C compiler could compile them.Consider what using Google would be like if it took 7 seconds rather than .7 seconds to respond.  Probably still useful, but a different user experience.We had many decades to be spoiled as far as program performance was concerned.  As our tools got more bloated, faster processors and more memory hid the true effect from most of us.  Now we have to rethink everything to make it parallel.  And complexity is enemy number one when making something parallel... Steve

On Sat, May 6, 2017 at 2:07 AM, Bakul Shah <bakul(a)bitblocks.com> wrote: Interesting analogy. I always think that function composition in lazy languages like Haskell reminds me of shell pipelines. I know that Doug swims in those waters; perhaps he will correct me if he feels that's a poor analogy? Carrying it forward a bit, however, I'm always amazed (and a little appalled) at how the kids always think that they invented Rock n' Roll. Ideas are continually recycled, but it's the rare recycler indeed who seeks out the prior art on rediscovering one of the good ones. "Look! You can filter and transform this data!" "...but I've been doing that at the shell nearly every day for the past quarter of a century...." "...this one goes to 11." - Dan C.

On 5 May 2017 22:33 -0700, from scj(a)yaccman.com (Steve Johnson): I would actually take that one step further: When you are writing code, you are _first and foremost_ communicating with whatever human will need to read or modify the code later. That human might be you, a colleague, or the violent psychopath who knows both where you live and where your little kids go to school (might as well be you). You should strive to write the code accordingly, _even if_ the odds of the threat ever materializing are slim at most. Style matters a lot, there. Sure, some code really needs to be heavily optimized, and might end up obscure for it. In operating systems, interrupt routines are probably one of the major candidates, but first stage bootloaders need their fair share of it too for other reasons. Tight loops or code that needs to be able to operate on lots of data quickly are other examples. But _most code isn't like that_ and can allow for some extra verbosity and some performance penalty if that means making it significantly easier to read and modify. That's not to say code should be verbose _just because_, but if you don't need ultimate performance, I say aim for readability and modifyability first; processing and memory usage second. You can always go back and optimize bits that turn out to need it. And of course, I find Doug McIlroy's shell snippet mentioned by Noel interesting, _because I did something very similar not all that long ago_; in my case, stringing together a bunch of shell tools to get a list of all URLs from an e-mail message for further processing. (Something like a composable _urlview_; if urlview had even had a "dump to stdout" option, that would have done all I needed it for.) Works like a charm, and isn't even terribly complicated. It could probably be done in a single line of perl or awk instead of piping through a grand total of six commands, but that's okay. And of course, stringing together a bunch of commands was the first approach that occured to me... -- Michael Kjörling • https://michael.kjorling.se • michael(a)kjorling.se “People who think they know everything really annoy those of us who know we don’t.” (Bjarne Stroustrup)

On 6 May 2017 09:09 -0400, from cym224(a)gmail.com (Nemo): True, optimized code can be difficult to read. But very little code really _needs_ to be optimized to that level, and when you come across it, you will generally know -- at least after running the application through a profiler. (I have to admit we do have it easy these days.) I've been working on optimizing some SQL code recently -- I know, that's not exactly operating system kernel C or assembler -- and it's amazing at times how the code can end up being both easier to understand _as well as_ faster to execute, if you just put some thought into what the code really needs to do. And not just minor speedups either, but wallclock runtime reductions on the order of _two to four nines_. I think there's been only one time in the last several years when I have really _needed_ to optimize code to the point that its workings were not obvious. -- Michael Kjörling • https://michael.kjorling.se • michael(a)kjorling.se “People who think they know everything really annoy those of us who know we don’t.” (Bjarne Stroustrup)

I am not a programmer. Almost four decades ago, my first computer was a TRS80 Model 1 with 16KB RAM. I spent one month disassembling and stepping through the Z80 code for the resident Microsoft BASIC interpreter. The entire thing fit in a 12KB PROM so it was originally written in assembly and tightly optimized. It was fascinating and extremely instructive. All these years later, I could probably still slap together a Z80 program if I needed to. Anyway, I reached one point in the assembly code that I simply could not understand. It seemed like a mistake, and I went through it again and again until I finally realized what it was doing. There was a branch/loop that jumped to the middle of a multi-byte machine instruction, so that branch had to be disassembled and stepped separately until it "synced" up with the other branch again. Maybe this is standard practice in programming (I don't know) but at the time I thought, what kind of evil genius devised this to save a few bytes of memory? --corey

On Sat, May 06, 2017 at 03:20:53PM +0000, Michael Kj??rling wrote: It strikes as being similar to Duff's device (1). Which is a niche thing but I still use that from time to time. Not to save memory, just because as a C programmer it seems pretty natural to do it. --lm (1) https://en.wikipedia.org/wiki/Duff's_device

On Sat, May 06, 2017 at 03:51:24PM +0000, Michael Kj??rling wrote: That's fine, I feel no need to argue about it. Seemed similar to me but I'm not the sharpest tool in the shed :)

On 6 May 2017 16:00 -0400, from usotsuki(a)buric.co (Steve Nickolas): You could do something quite similar on the 8086, which I am somewhat more familiar with. For example, in some kind of pseudo 8086 assembler, with $ denoting the value of the instruction pointer at the beginning of the current instruction: jmp short $+3 int 90h ; hex: cd 90 would almost certainly decompile to the above, if the decompiler doesn't barf when it can't create a jump target label mid-instruction, but it would certainly _execute_ as jmp short l1 db 0cdh ; dummy byte l1: nop ; hex: 90 resulting in a jump over one byte followed by a no-op instruction. Of course, invoking interrupt 90h would be a perfectly legal thing to do, assuming that your interrupt tables are set up correctly. If you want to mess with someone trying to figure out what the code is doing, write a (nonsense or meaningful) value to interrupt vector 90h before you do that. For example, assuming that it isn't being used, you could point interrupt 90h at the reboot jump location, which IIRC on the IBM PC would mean point it at FFFFh:FFF0h or absolute address FFFF0h. Anyone trying to actually execute the INT 90h instruction would see their computer reboot, but anyone actually executing the code would see little more than a CPU cache flush due to the jump. Or, something slightly more "useful" jmp short $+3 int 0 ; hex: cd 00 db 0 jz somewhere which, again IIRC, would clear the accumulator (AX) register because 0000 hex is XOR AX,AX, which as a side effect sets the zero flag because the result is zero, and then executes a conditional jump if the zero flag is set (which it will be). In other words, it _executes_ as jmp short l1 db 0cdh ; dummy byte l1: xor ax,ax ; hex: 00 00 jz somewhere but a naiive decompiler will see the CDh byte after the jump and take that as the first byte of the two-byte interrupt instruction. I don't know what 00 followed by the first byte of the JZ instruction would be, but probably a register/register XOR with a register store. Make it a far jump and you have a few extra bytes to play with, to befuddle anyone trying to figure out just what on Earth your code is really doing. The above, of course, is really just scratching the surface. With how many variable-length instructions the x86 has, with careful selection of opcodes and possibly use of undocumented but functionally valid combinations, I wouldn't be the least bit surprised if it's technically possible to write a 8086 program that does one thing when started normally, and something utterly different but still useful when started at an offset that results in execution beginning in the middle of an instruction. Bonus points if the first thing that program does is jump into itself _at that offset_. Now, the _work involved in doing so_, never mind maintaining it... Of course, I wouldn't do anything like the above in any real-world code base meant to run on modern systems unless obfuscation really _was_ a design goal. Honest. If I was tasked to write something that really needed to do something non-trivial in 16 KiB RAM on an original IBM 5150 PC or clone? I'd probably spend quite a bit of time in front of the whiteboard and with reference manuals before writing even one line of code... -- Michael Kjörling • https://michael.kjorling.se • michael(a)kjorling.se “People who think they know everything really annoy those of us who know we don’t.” (Bjarne Stroustrup)

On Sat, May 6, 2017 at 8:09 AM Corey Lindsly <corey(a)lod.com> wrote: This was extremely common back then. I had a friend who worked on a gas chromatograph project, names redacted here. It had a very advanced idea, a thermal printer. It would print a banner when it started. Getting that print to work, in the ROM they had, was a nightmare that involved all these tricks. When the "A" version came out, they asked my friend to have it print the A after the product number. His response: "NO". There was no way he could ever pick apart the crazy code that had printed out the startup banner so he could add an "A". The startup banner remained the same. Executing code as data in the early startup was also common in those days, and modifying that data and then rerunning it happened all the time. This is why the things like the Therac 25 happened ... https://en.wikipedia.org/wiki/Therac-25 Note the reference to "Cargo coding", reusing code you don't understand. In modern terms we call this software reuse and it's taught at all the best uni's. Google a package, pull it down, compile it in, done. Lest you think things are better now, Linux uses self modifying code to optimize certain critical operations, and at one talk I heard the speaker say that he'd like to put more self modifying code into Linux, "because it's fun". Oh boy. ron

On Sat, 6 May 2017, Larry McVoy wrote: A principle I've tried to follow over the years is "write code as though the next person to maintain it is an axe-wielding murderer who knows where you live". Apropos the above, I do confess to writing "clever code" from time to time (PERL is a beautiful language for such sins)... Of course, in the olden days it was the APL "one-liners" :-) -- Dave Horsfall DTM (VK2KFU) "Those who don't understand security will suffer."

On Sat, May 6, 2017 at 1:50 PM, Bakul Shah <bakul(a)bitblocks.com> wrote: The cool thing about this thread is that I learned two things: what tr -s does, and the Nq does for sed... Sadly, this doesn't work so well for text that isn't ASCII-7 english... Warner

On 7 May 2017 11:42 +1000, from noel.hunt(a)gmail.com (Noel Hunt): It does look like at least GNU coreutils 8.13 uniq is broken in that regard, which frankly surprised me. That version isn't _that_ old. $ uniq --version uniq (GNU coreutils) 8.13 Copyright (C) 2011 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>. This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Written by Richard M. Stallman and David MacKenzie. $ ( echo $'\u1234' ; echo $'\u2345' ; echo $'\u1234' ) ሴ ⍅ ሴ $ ( echo $'\u1234' ; echo $'\u2345' ; echo $'\u1234' ) | uniq ሴ $ -- Michael Kjörling • https://michael.kjorling.se • michael(a)kjorling.se “People who think they know everything really annoy those of us who know we don’t.” (Bjarne Stroustrup)

On 7 May 2017 08:58 -0600, from arnold(a)skeeve.com: I believe so (I don't recall seeing any other UTF-8-related weirdness for a very long time), but I would want to verify the behavior on a system that doesn't have a gazillion customizations accumulated over years _and_ has the most recent version of coreutils before I file an actual bug report. (It's not like Debian is ever bleeding edge, to begin with.) I'll see if I spin up a VM or three one of these days to check. -- Michael Kjörling • https://michael.kjorling.se • michael(a)kjorling.se “People who think they know everything really annoy those of us who know we don’t.” (Bjarne Stroustrup)

On Sat, May 06, 2017 at 09:18:57AM +0000, Michael Kjörling wrote: This can be, and is, taken too far. It is currently fashionable in the Ruby world to erect testing gantries that look like describe 'service' do it 'is listening' do expect(port 443).to be_listening end end and then drag in a couple hundred thousand lines of extra code to make this cuteness approximate instructions for the computer. It's all well and good to write for the person who comes behind you, but at some point it will break, and the closer you've flown to this sun the worse it will be for the poor sod who has to pick up the pieces. I'd take durable code I have to work to understand over fragile artwork any day. khm