[TUHS] Death by bug

[Jon Steinhart]

Theodore Y. Ts'o writes:
> On Sun, Jul 11, 2021 at 03:04:53AM -0600, arnold at skeeve.com wrote:
> > Ralph Corderoy <ralph at inputplus.co.uk> wrote:
> > 
> > > Given some devices are present in large numbers for many years in
> > > hospitals, and there's a lot of hospitals, an unnoticed bug could be
> > > steadily chipping away at its human overlords.
> > 
> > This is why I have purposely stayed away from jobs at companies doing
> > stuff like this. I know I don't write perfect code; I don't want to
> > be responsible for devices that can affect human life.  This is also
> > discussed in the new edition of "The Pragmatic Programmer", which I've
> > just finished reading. (Highly recommended.)
>
> We should never be depending on a human being able to write "perfect
> code".  Instead, we need to come up with processes so that imperfect
> code doesn't escape into production *despite* the fact that humans are
> fallible.  Such processes might include requiring unit tests,
> integration tests, stress tests, etc., requiring code reivews by a
> second pair of eyes, perhaps using formal proofs, having multiple
> implementations of critical algorithms, cross-checking the results
> from those independent implementations, and so on.
>
> The space shuttle used a number of these techniques.  It did *not*
> depend on super-human, Über-programmers.
>
> 						- Ted

In general I agree with you.  But the "we should never" ignores the fact
that we do.  It's a complicated problem.  A major component is that
our "legal" system allows companies to externalize their liabilities.
And it's difficult to come up a way to assign liability without severely
harming the open source world.  As I've said before, if your car crashed
as often as your Windows you probably wouldn't keep buying from the same
manufacturer.

Part of my preferred process is to have two software review teams.  One
has access to the source code, the other doesn't.  But of course this is
now often not possible when companies used closed-source components that
they purchase.

As someone who agrees with Feynman's dissent on the Challenger commission,
I think that incremental improvement is a good way to go.  But, we live in
an age where many folks would rather come up with something bloated and new
rather than build on what's come before.  That's making it more and more
difficult to distinguish between "full stack" and a landfill.

Jon