Itâ€™s Time for a Decent Mac Benchmark

Posted by Tonio Loewald on 06/1/07 in Apple, Articles, Hardware, Opinions

Xbench It seems like the first thing you’ll see in any review or comparison of new Mac hardware these days is XBench results. XBench seems to have become the the de-facto standard for Mac benchmarking, simply because their isn’t an alternative suite of tools. I’m writing this short article on a Mac Pro 2.66 GHz with 3GB of RAM. XBench is free (well, donationware), it’s easy to use, and it’s quick to download. It’s also almost completely useless.

Now now, before you get upset listen to my argument.

First of all, XBench is highly variable from version to version. My old dual 1GHz G4 scored ~150 in XBench 1.0 and ~43 in XBench 1.3 (the current version). My newish Mac Pro scores ~160 in 1.3. So, we might conclude that my new machine is about 4x faster than my old machine “overall”. But, it’s a little stranger than that. XBench’s algorithms (not just its calibration) varies from version to version. So, in its higher 1.0 score, the G4 achieved 4.91 Mops/sec and 425.78 MFlops/sec, but in its lower 1.3 score it achieved 5.05 Mops/sec and 827.4 MFlops/sec. The Mac Pro scored 16.54 Mops/sec and 3.61 GFlops/sec (or 3,610 MFlops/Sec).

So, ignoring the inter-version inconsistencies, the “overall” relative speed increase from G4 to Mac Pro is greater than the relative speed increase of the CPUs (a highly dubious result). Usually it’s the memory bandwidth, disk i/o, and graphics that drag a new system’s relative performance back down to Earth.

Here are some real-world actual task comparisons:

Rendering a fairly complex scene in Cheetah 3D (or any 3D package; I just happen to like Cheetah 3D). Mac Pro: 19.64s. Dual G4: 296.23s.
World of Warcraft Framerate on a busy day in Shattrah. Mac Pro: ~50fps. Dual G4: 5fps.
Gaussian Blur 30 pixel radius 6000×1000 pixel panorama in GraphicConverter using Core Image. Mac Pro: ~1s. Dual G4: ~8s.
Dark Strokes, default settings on 6000×1000 pixel panorama in Photoshop 7 (PowerPC native only). Mac Pro: ~4s. Dual G4: ~10s.

So when reviews in major magazines and websites quote XBench results and conclude that, say, the new 8-core Mac Pro doesn’t offer much of a performance boost, they need to give the lousy rating to XBench and perhaps the reviewer. I’m not saying the new 8-core Mac Pro is a must-have for any serious Mac user, but that it’s a serious machine that deserves proper reviews and real-world benchmarks.

Just for comparison, here’s Apple’s own benchmarks for the eight-core Mac Pro. Notice how they provide both artificial benchmarks (kind of like XBench’s processor benchmarks, only using an industry standard test) and representative application benchmarks (such as rendering time with Maya). Amazingly enough, these marketing benchmarks seem a lot more in line with real life experience than XBench’s. It’s a sad day when you’re better off reading the manufacturer’s marketing material than using third-party tools, but there you go.

What does everyone else use to benchmark their Mac?

del.icio.us tag this | digg this | permalink | ping

6 Comment(s)

Legend:

Guest

Article Author

Contributor

Damien said on

June 1st, 2007 at 9:56 am

I’d love to see someone create a series of standard tests that could be done by anyone and to collect them from different machines, so that we could really compare e.g. a PowerMac G4/533 (DA) against a 2ghz iMac (an upgrade I’m contemplating). Any suggestions on how to do the tests? I’d suggest using freely available tools (inc shareware / trialware), standardized files that can be loaded, and some Applescripts to launch & time the tests.

(subscribed to comments) Add karma

Matt J said on

June 1st, 2007 at 11:06 am

Would timing to completion of some kind of Automator workflow be an appropriate indicator?

Jaysen said on

June 1st, 2007 at 11:21 am

The problems with benchmarks are the type of calculations, the programmers “method” of performing calculations and the differences in hardware architecture. If we take a benchmark test that performs “large math” we may need different routines for each architecture (PPC, Intel Single/Dual/Quad/SMP) to optimize performance thereby reflect true possible speed. problem is that few software packages use separate routine optimizations (other than compiler) to maximize performance between architectures. So if we build a benchmark utility that is generic we penalize one or more architectures. On the other hand if we truly optimize then we are not comparing performance in a “real world” manner.

An example is dual core vs SMP. SMP is 2 chips, 2 sub systems. Dual core tries to emulate this but still gets bogged down in bus IO. So although we can use a similar function for the raw math the IO should be optimized for a dual core system. This would cause the SMP system to appear to not have a large benefit as the higher sustainable IO would never be measured. If we optimize for SMP then we penalize dual core by overloading the IO buss.

Also the “speed” rating of chips is not really an indicator of performance. Although a faster chip can cycle through calculations faster there are other factors like data size (16 vs 32 vs 64 vs 128) and bus speeds that have a larger impact. take a look at the initial offerings of almost any chip generation and you will find a speed overlap with the previous generation.

For those of you who remember the DEC “Alpha” series chips, they were 128bit

Jaysen said on

June 1st, 2007 at 12:37 pm

Someone should note teh max char limit on the comments…

What I was saying is that the 128bit

Tutor said on

June 3rd, 2007 at 9:43 am

You are right about Xbench. It doesn’t even produce the same results on the same machine - useless.

I favor timing real applications these days, too.

imajoebob said on

June 11th, 2007 at 12:38 am

I’ve noticed that the speed of your hard drive seems to influence all the scores, not just the disk rating. Makes me wonder just how discrete the individual tests are.