acm-header
Sign In

Communications of the ACM

ACM News

New Supercomputer Benchmarks Overdue?


View as: Print Mobile App Share: Send by email Share on reddit Share on StumbleUpon Share on Hacker News Share on Tweeter Share on Facebook
Allan Snavely

Allan Snavely is associate director of the San Diego Supercomputer Center (SDSC) at the University of California, San Diego.

The yardstick used to identify the world’s best-performing supercomputers on the Top 500 list–the Linpack Benchmark–no longer reflects "real-world usage" and ought to be replaced with a new metric.

So says Allan Snavely, associate director of the San Diego Supercomputer Center (SDSC) at the University of California, San Diego. He proposes, instead, a "data motion capacity metric" that measures a supercomputer's overall ability to help researchers solve real-world problems. Instead of comparing supercomputers by their fastest calculation speed–typically measured in the number of floating point operations per second (FLOP/S)–he suggests a measurement that weighs DRAM, flash memory, and disk capacity according to access time as measured in computer cycles.

"Processors have become increasingly faster, but the performance of real applications haven't kept pace due to the slower data-fetching and data-movement aspects of the machines," he explains. "In fact, disks are in a sense getting slower all the time as their capacity goes up but access times stay the same."

Snavely contends it’s not really helpful to have rankings that "only reflect the features that have been improved to the point where they are no longer the bottleneck. That metric is saying a given supercomputer is the best around, but it’s not really."

Horst D. Simon, deputy laboratory director at the Lawrence Berkeley National Laboratory, says he wholeheartedly agrees with Snavely’s efforts and that "as we move forward, it is clear that FLOP/S is not the right way to measure the performance of high-end machines."

 

Horst D. Simon UC-Berkeley's Simon Horst 
Simon recalls that FLOP/S was a very important characteristic of computers in the 1960s and 1970s, but things have changed, he says, adding it is critically important that the focus be on the total system architecture–which includes FLOP/S but also, as Snavely points out, data access, and amount of memory, flash, disk space, and so on.

Simon looks forward to reading a white paper on Snavely’s proposal when it is written, "as, I’m sure, will the rest of the supercomputing community. Then," he adds, "the next step will be for Allan to get his proposal accepted, which will take time. First, he will need to get buy-in from the community, then he will need to have his benchmark implemented on at least a dozen very different architectures, report on what has been found, and then involve the community in discussions. It's easy to propose a benchmark, but actually getting it accepted takes a lot of work over multiple years."

Meanwhile, the SDSC is about to roll out its new Gordon supercomputer designed more for data-intensive supercomputing than speed. Snavely says its capabilities for accessing data on disk are unprecedented but admits that it will "be a way's down on the Top500 list–maybe around #40–but that's only because of the skewed metric that list uses."

In order to generate interest in his proposal, Snavely will speak at the SC11 Supercomputing Conference in Seattle on Nov. 15, approximately the same time as the Gordon rollout.

"Then, early next year, I hope to publish a white paper along with a benchmark and a simple metric, a test that people can run on their supercomputer to measure its data-moving capacity," he says. "I’m hoping people will start sending us their data which we can compile on a new Web site.

"What I would argue," he says, "is that, if it weren’t for the Linpack Benchmark–which reflects very little of the real world and is a very extreme and tailored benchmark for measuring only the FLOP/S rate–we would probably have a much more interesting and useful list of supercomputers by capability."


Paul Hyman was editor-in-chief of several hi-tech publications at CMP Media, including Electronic Buyers’ News.
 


 

No entries found

Sign In for Full Access
» Forgot Password? » Create an ACM Web Account