[NCLUG] The kiss principle and Operating Systems

[John L. Bass]

Eric S. Raymond’s work, The Art of UNIX Programming, offers lots of
insights. To quote:

       [http://www.faqs.org/docs/artu/ch01s06.html#id2878022]

       Rule of Parsimony: Write a big program only when it is clear
       by demonstration that nothing else will do.

       “Big” here has the sense both of large in volume of code and
       of internal complexity. Allowing programs to get large hurts
       maintainability. Because people are reluctant to throw away
       the visible product of lots of work, large programs invite
       overinvestment in approaches that are failed or suboptimal.

In the UNIX tradition this had been exemplified by using the 90/10
rule to pickup 90% of the functionality with 10% of the work, and
ignore the remaining 10% which will in the long term become 90% of
the work, or more.

The biggest problem with OSS, is that it doesn't have any good way
to keep the kitchen sink out, and with lots of contributors, generates
"overinvestment" in a large solution growing without controls.

The UNIX standards committee struggled with this greatly, with many
well reasoned "improvements" to add to the core standard. That file
locking ended up in the standard as the only non-USL code, was only
because there was high demand for the feature, and no workaround. I
had already released it in the public domain, and was widely included
in most vendors releases and required by RM Cobol (and other business
applications) for database locking.

   http://www.bitsavers.org/bits/Interdata/32bit/unix/univWollongong_v7/tar/1674_edVIIupg2.3binSrc_Apr83/src/sys/lockf.c

Comments by Nicklaus Wirth in 1978 regarding the standards process
for Pascal, offered some insight and justification for keeping it
lean. Nicklaus lamented that the most difficult task in the process
was keeping every well meaning extension out of the language to keep
the language simple and easy to use. By reference, IBM's PL/1 was the
kitchen sink example. [If somebody knows where the exact quote is
online, I'd appreciate it ... may be in the Modula-2 papers -- John]

At SCO in the early 1990's I did a number of performance projects
centered around both memory and disk. After teaching an internal
brown bag lunch program on performance, one team went off and
implemented several ideas I presented to fix the cache aliasing
problem. They did a utility that would profile a large application,
and using the profile data would reorganize instructions to mimimize
cache faults. The results were so remarkable, that they released it
to customers at SCO Forum later that year. No coding changes, and
many large programs saw 3-8X performance gains. Doing something like
this for Linux is a possible short term band-aid, made difficult by
dynamically loadable kernel modules and libraries at the user level.

By the second release of System-V the hand writting was on the wall
that this was growing out of control. Two years later, it was clear.
System working sets could only be managed by the largest machines
built, and usability on machines as little as 3 years old was getting
difficult.

The exponential growth of the linux kernel over the last 10 years
is equally frightning, and similar observations apply. I have several
machine clusters as my "home system" ... 32 dual processor Tyan PIII
machines (1.0 to 1.4GHz) by Rackable Systems and a dozen dual/quad
SR870 based Itanium2 servers with a mix of CPU's. This is supported
by multiple gigabit ethernet and Fibre Channel subnets, with several
dozen scsi/fc spindles providing several terabyte RAID and JOD arrays.

In both large clusters there is a clear non-linear performance difference
between machines strickly corrilated to L2/L3 cache sizes independent
of CPU speed and available memory. There is also high variability in
run to run for the same workload simpily due to physical memory
allocation order. This has gotten worse from FC4 thru F7. 

Given that it takes 3-8 years to mature a new operating system, and
the rate that Linux is bloating, I'm pretty sure a lot of folks need
to start working on a next generation kernel designs.

I've been running UNIX as a home system since DEC LSI 11/23's came
out in the late 1970's, and got my "legal" UNIX license shortly
after Microsoft started selling PDP-11 Xenix licenses in 1981.

This winter I will be heating my home with these two clusters while
taking on a couple computationally difficult projects and continuing
some personal kernel development in distributed filesystems and cluster
management. There is a good chance that I will update and port the
10 year old ExOS project to both clusters, but I'm still doing my
homework to see if there is a better starting point for a light wieght
replacement for Linux. Pointers welcome.

The 8X improvements that ExOS exhibted 10 years ago, are likely to
be much larger today given the increasing spread between cpu clock
rate and memory cycle times. As a UNIX Kiss guy, there is a lot that
makes sense about their libos approach.

If somebody has some good HPC cluster ideas, they are certainly welcome
to contribute and join the fun.

John