[TUHS] Benchmarks....

[Bakul Shah]

> On Feb 16, 2022, at 2:42 PM, Larry McVoy <lm at mcvoy.com> wrote:
> 
> On Thu, Feb 17, 2022 at 09:27:39AM +1100, Dave Horsfall wrote:
>> Indeed, when a lot of compilers recognised the Sieve of Eratosthenes being 
>> used and optimised for it...  Wasn't all that long ago that vehicle 
>> manufacturers also started doing the same thing :-)
> 
> Indeed.  I hated when people cheated because benchmarks should teach you
> the capabilities of the machine being benchmarked.  They aren't useful
> if people cheat.
> 
> When I did LMbench, the rules were "cc -O2" and you could not link with
> any benchmarking libraries and you could only report a result if you
> reported all results.
> 
> I had a beef with this guy: https://www.cs.jhu.edu/~shap/ because he
> had some toy OS that didn't have a VM system, didn't have networking,
> didn't have much of anything, but boy oh boy, did it context switch
> fast (because there was no context to speak of).  He reported those
> numbers in blatant violation of my rules and I had to escalate to 
> get him to stop.  The bummer is he is a smart guy, capable of good
> work, there is no need to cheat.

My guess is, it was EROS, based on KeyKOS (or GNOSIS), a capability
OS. It used the VM hardware to enforce capabilities so I imagine
it did have a VM system!

http://flint.cs.yale.edu/cs428/doc/eros.pdf

The paper does say "microbenchmarks ... inspired by those of 
lmbench". My guess is Jon Shapiro wanted to show off his system's
nimbleness. Should not be seen as an apples to apples comparison
with commercial Unix systems.

[Don't want to reopen a microkernel/capability system discussion
 here though]