Futuremark Delists Samsung and HTC Android Devices for Cheating 3DMark

[wraggster]

Benchmarks are serious business. Buying decisions are often made based on how well a product scores, which is why the press and analysts spend so much time putting new gadgets through their paces. However, benchmarks are only meaningful when there's a level playing field, and when companies try to 'game' the business of benchmarking, it's not only a form of cheating, it also bamboozles potential buyers who (rightfully) assume the numbers are supposed mean something. 3D graphics benchmark software developer Futuremark just 'delisted' a bunch of devices from its 3DMark benchmark results database because it suspects foul play is at hand. Of the devices listed, it appears Samsung and HTC in particular (http://hothardware.com/News/Futuremark-Delists-Android-Devices-Allegedly-Cheating-3DMark-Samsung-and-HTC-Ousted/) are indirectly being accused of cheating 3DMark (http://slashdot.org/story/13/07/31/041250/samsung-caught-boosting-galaxy-s4-benchmarks) for mobile devices. Delisted devices are stripped of their rank and scores. Futuremark didn't elaborate on which specific rule(s) these devices broke, but a look at the company's benchmarking policies reveals that hardware makers aren't allowed to make optimizations specific to 3DMark, nor are platforms allowed to detect the launch of the benchmark executable unless it's needed to enable multi-GPU and/or there's a known conflict that would prevent it from running.

http://hardware.slashdot.org/story/13/11/26/0257221/futuremark-delists-samsung-and-htc-android-devices-for-cheating-3dmark