Performance optimization on the server, or even on the desktop, has been a mostly solved problem for many years. But mobile devices are just now getting the attention they deserve from hardware and software vendors when it comes to performance optimization and benchmarking tools. Of the many new and updated tools arriving this spring, the most vendor-neutral is surely AndEBench 1.0, from the non-profit Embedded Microprocessor Benchmark Consortium (EEMBC).

AndEBench 1.0 was built by this vendor-backed consortium because the many Android CPU benchmarking tools out there are not transparent, said Markus Levy, president of the EEMBC. “The EEMBC is a benchmarking organization that’s been doing this for over 15 years,” he said.

“Originally, we were focused on the CPU side of things. Traditionally, we have C-coded benchmarks that you run through a compiler and run directly on the machine. This was targeted at embedded systems. The processors are still there, but they’re moving to Streaming SIMD Extensions (SSE). The only way to benchmark SSE is by taking it to the system level, and you can’t write C code everyone can run for that without porting the code. If you port the code, then it’s not comparing apples to apples.”

But in the world of Android, benchmarking can still be done at both the native and the Java levels. In fact, said Levy, AndEBench runs against both by default. And while this is not new for the Android space, he said AndEBench is the first truly committee-built benchmark for the platform.

That means the device manufacturers themselves decided how these benchmarks would run, making AndEBench the Switzerland of benchmarks. That’s an important distinction, because Levy said there’s no lack of benchmarks in the Android Marketplace. “The biggest difference is that we develop this benchmark through committee,” he said.

“It’s been worked on by members like Intel, ARM, nVidia, Qualcomm, Dell and Motorola, who have contributed what they think this benchmark should be to make it fair and equitable. That is our fundamental differentiator. The other thing is that when you look at the plethora of Android benchmarks out there, it’s clear that anybody can write a benchmark and put it on the marketplace, but how do you know what’s in it? Even Qualcomm has their own, but nobody knows what it does.”

AndEBench is freely available in the Android Marketplace today, with advanced features and source code available for a $1,500 license fee.

Not alone on benchmarks
The EEMBC is not the only group with a new performance analysis tool available for Android and other mobile devices. At the Game Developer Conference in San Francisco last week, the three largest chipmakers for mobile devices each had their own CPU/GPU performance analysis tools to show off.

Joshua Doss, graphics developer for performance tools at Intel’s developer products division, showed off the next version of the free Graphics Performance Analyzers, also known as GPA. This tool allows developers to feed live data out of the CPU and GPU to track loads across processes. With some special ties to Intel’s graphics chips, the tool can even modify running applications and swap out levels of rendering to reveal wireframes and collision points.

Perhaps the most interesting new development for GPA is that Intel will be releasing a version of the tool that works with Android devices. While no release date has been set, Doss showed a version where he was able to read out performance information live from a running Android game with very little performance slowdown.

Imagination Technologies showed a similar tool, though this one was optimized for artists working with devices that use Imagination’s chipsets. The PowerVR graphics intellectual property, which Imagination owns, is used in Android and iPad devices alike. Thus, PowerVR and the accompanying SDK now include pre-rendering pipelines to allow graphic artists to see what their finished work will look like without having to run files through slow rendering processes.

Imagination is also pushing its own PowerVR-based profiling tools, which allow developers to glean data from running CPUs and GPUs.

But the biggest fish in the mobile pond is, without a doubt, ARM. At GDC, ARM too showed off its own free CPU/GPU profiling tool, which looked and behaved much the same as those from Imagination and Intel. The key difference here was that ARM’s tool lives inside of Eclipse, and that it’s also been updated to be compatible with ARM’s Mali graphics hardware.