[NCLUG] parallel processing users?

[mike loseke]

 I support a cluster of machines running Linux on 3 different architectures
as well as Sparc Solaris. The work being done on this cluster uses EDA tools
which include quite a bit of simulation and regression type work. In
general, none of the tools in use here (100+ tools from 35+ different
vendors) are capable of traditional parallel or threaded operation as much
of what they are doing is too linear in nature or based upon timing.

As such, we require a large amount of horsepower but not so much like what
large monolithic systems or traditional clusters use. Instead, we use a
batch queueing system from a company called 'Platform Computing' called LSF,
or Load Sharing Facility. At its most simple config, it's just a batch
system, but it's very good at handling thousands upon thousands of jobs at a
time. It lets us treat any individual member of the cluster as a simple node
which can be removed at any point and only minimally affect performance.

Some newer EDA tools are doing some interesting things like splitting up
large jobs into groups of smaller jobs and submitting those to the queueing
system, some even going so far as to specifically working smoothly with LSF.
The submitting process waits for the spawned child jobs to finish and then
assembles the finished product, whatever it may be. In this same cluster, we
have jobs that run for weeks on end as well as some that run in bunches of
several hundred for seconds each at a time. For what we do, it's a great
system.

So from this perspecitve, individual chips can't ever be 'fast enough', but
more and more our tools and infrastructure are supporting methods to take
advantage of more CPUs.

On 10/14/05, Matt <rosing at peakfive.com> wrote:
>
> Anyway, performance is what it's all about. I'm interested in finding
> people that use parallel processing because a single processor can't
> go fast enough. Could be distributed jobs, or tightly communicating
> models or data mining or computational biology or using specialized
> processors or who knows what. I'm open to anything. It just seems
> that now, since dual core processors are coming out, it might be the
> case that making faster chips is harder than parallel chips, so
> parallel processing might become more important.
>