Array Processors and UNIX

[Peter H. Berens]

I would like to thank those who mentioned my name and the name of my company
in regard to UNIX software for array processors.  I would, however, like
to address some of the points made in many of the recent submissions and
perhaps shed some light on a few areas.

I would first like to say that all the observations made in all the
previous articles are correct.  I have experienced almost all of the same
difficulties myself in the over ten years I have been a user of FPS
array processors.  However, what most of the authors fail to mention
is how OLD their experience, array processor, and/or software are. I think I
can demonstrate here that almost all of the specific problems presented
thus far are more of a history of the evolution of array processors and of
FPS as a company, rather than the problems that potential users of today
are likely to face. Thus, I would like to tackle each issue on a point by
point basis below:

First to respond to Mike Ressler's original question:  What array processors
run under UNIX:
	Currently I am not aware of any array processor manufacturer that
	provides software that will run under UNIX.  To the best of my
	knowledge my company (APUNIX) is the only source for UNIX array
	processor software.  We currently support the FPS 5000 series
	of array processors.  They range in speed from 8 megaflops (millions
	of floating point operations per second) to 60 megaflops.  The price
	is somewhere between 40-90K for the hardware depending on speed.
	We also support the predecessors of the FPS 5000, the AP-120B
	and the FPS-100.

	There ARE other manufacturers of floating point array processors:
	CSPI, Analogic, Star, Numerix, Sky Computers, and CD&A.  There
	may be even more I have not yet heard of, but be careful when
	looking at their literature to make sure it is a real floating point
	and not either block-floating point or integer.  For reasons I
	won't go into here you want to avoid these other types of AP's.
	FPS and CSPI are the oldest of the array processor manufacturers.
	FPS has always been very successful in marketing against CSPI and
	thus has generally been considered the leader in the field of
	array processors.  I too have heard the rumors about CSPI and
	their support of UNIX with their mini-map array processor.  I would
	only caution people to check it out throughly before buying it.
	Insist on talking to people who already are running it under UNIX.
	And if they should convence you to be the guinea pig, make sure
	you get one hell of a discount on the hardware (as in free).
	I have approached all of the other manufacturers about having my company
	providing UNIX support and have meet mostly with cold receptions.
	We are still open to providing UNIX support for any new array
	processors regardless of manufacturer if suitable terms can be arranged.

Why don't the manufacturers support UNIX?
	J. Wong shed some inside light on this question, but I would like
	to add a little here as well.  Almost without exception the array
	processor manufacturers write their support code in Fortran.
	Given the non compatibility of Fortran between the major hosts
	(DEC, IBM, etc.) they already have a VERY hard time keeping the
	support going for the manufacturer's supported operating systems.
	These companies tend to have very limited software engineering
	resources and the talent they do have tends to be very specialized
	(remember they have to write device drivers for EACH operating system
	they support, and most aren't as easy as UNIX).
	It is often a choice for them between supporting a new piece
	of hardware they may want to introduce or adding more supported
	host operating systems to their present products.  Given the
	recent boom in VLSI array processor chips, they all are focusing
	on new hardware rather than expanding their current market.
	Also given the horror stories of the UNIX f77 compiler and its non
	compatibility with VMS Fortran for example, you can see why they
	would think hard before tackling such a project.  Also I am not
	sure the UNIX community would want them to.  Even though they may
	clean up their Fortran to run under f77,  do you want to use software
	tools that still provide an interface like VMS?  They would hardly
	be willing to go out of their way to customize the software for
	UNIX and thus have to redo all their documentation and training
	programs.

	You may ask is UNIX customization important.  I think most of our
	customers would tell you quite definitely yes.  They like to be
	able to run part of their code through M4, for example, and then
	pipe the rest on to some other part of our software.  They also
	like to be able to run the software out of makefiles and have all
	the appropriate options available as switches like the rest of UNIX
	commands.  And as James Matheson points out, they sometimes are
	unwilling to give out source whereas a company such as ours would
	never try to sell to the UNIX community with anything but a full
	source license.

Why doesn't the potential UNIX user just buy one of these things and write
the device driver himself?
	Well, if it was ONLY a device driver I might agree on this point.
	(Although there are some key performance issues where home brewed
	device drivers might not fair so well).  The main fly in the ointment
	is that array processors come with very HUGE software packages called
	program development software.  Remember these are not just fancy
	controllers, these are complete independent perhiperal computers.
	They have their own instruction language and libraries of microcode.
	Thus you need an assembler, linker, loader, simulator, and higher
	level language interfaces.  You also need a ton of host-AP interface
	routines.  To provide a satisfactory product for UNIX users, we have
	had to rewrite ALL of the above in C and optimize for UNIX.
	This was a process that took several man years of work to evolve
	to its current state.  Its not the sort of thing you want to get
	into unless you either have no choice or are going to make a product
	out of it.

Who are these guys at Stanford?
	About ten years ago I started working with Serial #2 of the first
	FPS array processor.  At the same time the site I was at received
	one of the early Bell releases of V6 UNIX.  Due to the conversion
	problems of the software we ran the AP under RT-11 for a little over
	a year.  Then at Stanford we found two people who were in a similar
	situation as ours.  Their names are John Newkirk and Rob Mathews.
	John and Rob came up with a preliminary version of the software
	than ran under the once infamous Princeton Fortran Compiler and/or
	the CULC Fortran Compiler.  Then the three of us began the job
	of rewriting the software and evolving it into a real UNIX product
	(I did most of the rewritting of the monster Fortran programs to C,
	they worked mainly on the driver and support library).
	During the early years, John and Rob marketing the package as a
	company called Peninsula Research.  However, as they became involved
	heavily in other things, support became a real problem for them.
	At that time they turned the complete package over to me and
	APUNIX was born.  Thus Stanford or Peninsula Research no longer
	provide any UNIX support software for array processors.  Unfortunately,
	due to poor records, I was unable to get in contact with many of
	the sites they had distributed the software to.  Ballistic Research
	Labs is one of these sites.  Since the time APUNIX has been responsible
	for the software,  it has evolved into an entirely new product with
	most of the performance issues raised by Ron @ BRL corrected.

Why buy an array processor and not a vector processor?
	It is my opinion that array processors still offer a
	significantly higher performance/cost ratio than even the new
	vector processor machines (e.g., mini-Crays) that are becoming
	available.  One must remember that the new array processors are
	in the 60-300 Megaflop range and not the 10 megaflop range the
	previous submitters are basing their experience on.  The array
	processor offers you the advantage of actually coding for the
	intrinsic parallelism and pipelining inherent in floating point
	hardware.  The vector processors really hide this fact and to
	do so as efficiently as they do is probably where a lot of the cost
	goes.  The advantage the vector processors have is their compilers,
	however a good microcoded library routine that performs the same
	function on an array processor will still be more cost effective.
	There is only one good compiler available for an array processor
	that I know of, and that is for the FPS-164 (the 64 bit brother of
	the FPS-5000 series).  The FPS-164 has a speed range of
	10-300 Megaflops but a price tag that matches (>300K).  We are
	currently negotiating for the UNIX support for the FPS-164, but
	no deals have been sealed as of yet.

Now onto the FPS array processors specifically:
Are they hard to program?
	Yes and no.  The Yes part is in generating your own microcode
	for the AP.  It has a highly parallel and pipelined architecture
	that is reflected in the complexity of its assembly language.
	However, I estimate that less than 10% of our customers actually
	do microcoding of their own.  The major advantage of FPS is the
	extensive math libraries that exist of canned routines in the
	AP's microcode.  This allows most people to simply pick out
	one or more of these functions and link them together in an
	appropriate manner to solve their problem.  Most people are very
	happy with the results and DO achieve substantial (5-10 times
	a VAX 11/780) with just this level of effort.  Given the UNIX
	philosophy that assembly language is a dead language, I wont say
	much about writing assembly language routines for the AP except
	that it is not as bad as it seems.

	The major problem in most user's eyes is that there is no compiler.
	That is to say, you simply can't take the code you have been running
	on your vax and compile it and have it run on the AP.  You must
	alter your code slightly to perform the following tasks: a) transfer
	the data to and from the AP's memory and b) call the appropriate
	AP program(s) (from the libraries)  to perform the desired tasks.
	FPS does have a Fortran compiler for the 5000's, but the current
	version is so bad they have a hard time selling it to any customers,
	due to its poor code generation.  Thus we do not provide it with our
	UNIX software package.  There is suppossedly a better compiler from FPS
	soon to be available. (It is actually being done by a group called
	Systems Software Factors in England, also known as the Toast People.)
	We do intend to provide this if it indeed proves to be a viable
	alternative.  (The advantage to us is that some years ago I managed
	to help convince the Toast people that C was a better language to write
	their compiler in than Fortran!)

Are they UNIBUS hogs and is there a lot of system overhead associated with
their use?
	Yes and no.  In most cases they only are if you let them
	be.  The most serious overhead is in transferring the data back
	and forth between the host and the array processor (as most array
	processors have their own memory, although some may argue shared
	memory approach may have some advantages, normally the complexity
	of dealing with it in a multiuser virtual memory system and the
	overhead of accessing it over the UNIBUS (in the case of vaxes) I
	feel negate any benefits). In most of the cases where I have seen 
	interface bandwidth to be a problem, it is mostly
	due to very little thought being given to the problem.  Generally
	you can arrange it so you transfer an initial set of data to the AP
	do one or more operations (hopefully more) and keep intermediate
	results in the AP and then just bring back the final result.  Since
	all the new FPS 5000's come with a minimum of 256K words, storage
	of data and intermediate results is not a problem.  If for example,
	you transfer two arrays and do a vector add and transfer them back,
	and the arrays are only five elements long then forget it.  The
	overhead will kill you.  But if the arrays are > 20-100 elements
	and you have to do a few more things besides one vector add on
	the data before you have solved your problem, then you will start
	to notice the blazing speed of the AP.

	If your applications were extremely i/o intensive, I might advise
	you to be to carefully analyze the costs of the data transfers
	before buying an AP.  But even in such applications such as image
	processing we have customers who are very happy with the AP and
	don't see the i/o bandwidth as a real problem either in terms
	of performance or system load.

	System overhead is an area where the design of the device driver
	plays a key role.  Older versions of our device driver had a
	severe problem in this regard.  However, over the years we have
	rewritten the driver many times using many different strategies
	to try to minimize this.  We feel we that we have now incorporated
	a method that makes the actual overhead time comparable to that
	which is achievable with VMS.

Is FPS non-responsive to hardware problems and do they still ship broken
hardware?
	One must remember that in the past 10 years, FPS has grown from
	a garage operation to a multi-million dollar company.  During that
	time they necessarily suffered many growing pains.  I am not trying
	to defend them, as I too cursed their name for many years when
	their non-responsiveness caused me many hours and weeks of down
	time.  We HAD to maintain the hardware ourselves for many years
	because FPS support was so bad.  But one must keep in mind that
	this is not typical of their behavior TODAY or in the last three
	to four years.

	The problems Ron @ BRL points out with the interface problems
	that required hardware ECO's to fix have not been in existence
	for many years.  It is a lot like getting Rev A of a disk controller,
	I don't think its fair to tell people not buy the product because
	of problems with Rev A when the company is shipping Rev Z today.
	Malcolm @ Purdue also points out that he must keep his AP on its
	own UNIBUS adapter.  This was also true for about Rev A-C (actually they
	only had to be at the END of a busy UNIBUS), but again
	we are on Rev Z today.  All of the AP's I have seen shipped in the
	last three years have NO interface problems even when on a UNIBUS
	with disks, dz's, dh's,  and all other perhiperials possible
	(with the possible exception of a Benson/Varian printer plotter,
	and I still firmly believe the problem to be in the Benson/Varian
	interface).  And most of these AP's have been installed without
	any hardware problems.

Will FPS install or support the array processor under UNIX?
	This is a real grey area.  In most of the cases I am aware of
	FPS has been willing to install the hardware and check it out
	with our diagnostics under UNIX.  I really don't quite understand why
	Chris Moore @ AMD has had this particular problem.  His neighbor
	in Sunneyvale and uucp partner, Resonex, got an AP at almost the
	exact same time.  They were able to convenience FPS to install it
	with our diagnostics, but apparently he was not.

	Hardware maintenance is largely dependent on the local FPS support
	people.  Most I have dealt with have been willing to look at the
	results of the diagnostics running under UNIX and then fix the
	problem.  Once one fires up the diagnostics, they look exactly the
	same as they are under VMS.  The best thing to do is have
	them running when the FPS service engineer
	walks in the door and don't mention the word
	UNIX.  The same is true when calling FPS in Portland for assistance.
	They are slowly warming up to the idea that UNIX is not a bad word,
	but it still is not a good idea to make an issue out of it when
	trying to get their help.  We, of course, are willing to supply
	all the support and maintenance for the software as may be necessary.

Are people who have the APUNIX software and FPS array processors happy?
	I will have to let my customers speak for themselves.  But at least
	from what I can determine, most seem to be pleased the performance
	of the software/hardware combination and find the AP to provide
	them with quite a bit of bang for the buck in terms of computational
	ability.  I have had rough times with some customers, where there
	may have been a period of dissatisfaction.  But we have tried to work
	with each of them and I don't think there is one in the end who
	has not been happy.  We don't have any customers I would be hesitant
	to give as a reference.

	Most of our customers have been worth their weight in gold to us.
	Malcolm @ purdue is one of those rare sites you can send experimental
	software and get constructive criticism back.  He has also shared many
	of his own enhancements with us.  George Tomasevich @ Bell Labs
	is at a site where we encountered a particular configuration of an
	AP we had never seen before.  It caused our software to misbehave
	in mysterious ways and they had the patience wait a few weeks
	and give us access to their system while we tracked down and
	corrected the problems.  Glen Adams @ mit-lincoln labs also
	contributed grately when the software needed to be retrofitted
	to run on a V7 PDP-11 after many years of VAX-ination.  I mention
	these people in particular (there are many others we are equally
	as grateful to  as well) because they have given us good
	recommendations on the net, but yet I know they were by far not
	our more typical smooth installations.  I only hope that this
	speaks highly of the level of customer satisfaction we have been
	able to achieve and the support and commitment we have to the software.
	I should also mention that the FPS-5000 is a brand new product, and
	as such our current FPS-5000 sites have been quite patient while we
	scramble to bring up a bunch of new support software for the additional
	co-processor capability that will hopefully allow them to run at a rate
	of 30-50 times that of a VAX 11/780.  They have been kind enough not
	to point our this temporary shortcoming on the net.

What is the connection between APUNIX and FPS and the difference between
their software?
	There is really no connection between the two of us expect that
	we happen to provide the UNIX support software for their hardware.
	Over the years, our relationship with FPS has been somewhat
	cyclic.  However, in recent years that have been quite supportive
	of our efforts and have provided us with a complete distribution of
	their latest software and keep up updated on all hardware and
	software bug fixes.  All sales and marketing are still done
	separately however.  They take no responsibility for our software.
	You must by the hardware from them and then on a separate P. O., buy
	the software from us.  Its a lot like buying a VAX from DEC and the
	UNIX license from Bell.

	With the exception of support for the old Fortran compiler, I know
	of no major feature of the FPS software they we do not support in
	our UNIX version.  In fact, we offer many additions such as a
	C syntax higher level language interpreter for linking together
	math library routines.

For more information, please feel free to contact us or give us a call.
I will be glad to tell you anything I can about array processing under
UNIX even with respect to products we don't support.
I know this message has gotten quite long, thank you for your time.

					Peter H. Berens
					Apunix Computer Services
					1380 Garnet Ave., Suite E-292
					San Diego, CA  92109
					(619) 484-0074
					Telex: 4997136 APUNIX
					...!decvax!ittvax!dcdwest!phb
					...!ucbvax!sdcsvax!dcdwest!phb