Performance optimization on the server, or even on the desktop, has been a mostly solved problem for many years. But mobile devices are just now getting the attention they deserve from hardware and software vendors when it comes to performance optimization and benchmarking tools. Of the many new and updated tools arriving this spring, the most vendor-neutral is surely AndEBench 1.0, from the non-profit Embedded Microprocessor Benchmark Consortium (EEMBC).

AndEBench 1.0 was built by this vendor-backed consortium because the many Android CPU benchmarking tools out there are not transparent, said Markus Levy, president of the EEMBC. “The EEMBC is a benchmarking organization that’s been doing this for over 15 years,” he said.

“Originally, we were focused on the CPU side of things. Traditionally, we have C-coded benchmarks that you run through a compiler and run directly on the machine. This was targeted at embedded systems. The processors are still there, but they’re moving to Streaming SIMD Extensions (SSE). The only way to benchmark SSE is by taking it to the system level, and you can’t write C code everyone can run for that without porting the code. If you port the code, then it’s not comparing apples to apples.”

But in the world of Android, benchmarking can still be done at both the native and the Java levels. In fact, said Levy, AndEBench runs against both by default. And while this is not new for the Android space, he said AndEBench is the first truly committee-built benchmark for the platform.

That means the device manufacturers themselves decided how these benchmarks would run, making AndEBench the Switzerland of benchmarks. That’s an important distinction, because Levy said there’s no lack of benchmarks in the Android Marketplace. “The biggest difference is that we develop this benchmark through committee,” he said.

“It’s been worked on by members like Intel, ARM, nVidia, Qualcomm, Dell and Motorola, who have contributed what they think this benchmark should be to make it fair and equitable. That is our fundamental differentiator. The other thing is that when you look at the plethora of Android benchmarks out there, it’s clear that anybody can write a benchmark and put it on the marketplace, but how do you know what’s in it? Even Qualcomm has their own, but nobody knows what it does.”

AndEBench is freely available in the Android Marketplace today, with advanced features and source code available for a $1,500 license fee.

Not alone on benchmarks
The EEMBC is not the only group with a new performance analysis tool available for Android and other mobile devices. At the Game Developer Conference in San Francisco last week, the three largest chipmakers for mobile devices each had their own CPU/GPU performance analysis tools to show off.