Sleaze: Nearly all Android device makers cheat on benchmarks

“We started piecing this data together back in July, and even had conversations with both silicon vendors and OEMs about getting it to stop,” Anand Lal Shimpi and Brian Klug report for AnandTech.

“With the exception of Apple and Motorola, literally every single OEM we’ve worked with ships (or has shipped) at least one device that runs this silly CPU optimization. It’s possible that older Motorola devices might’ve done the same thing, but none of the newer devices we have on hand exhibited the behavior,” Lal Shimpi and Klug report. “It’s a systemic problem that seems to have surfaced over the last two years, and one that extends far beyond Samsung.”

“As we mentioned back in July, all of this is wrong and really isn’t worth the minimal effort the OEMs put into even playing these games. If I ran the software group at any of these companies running the cost/benefit analysis on chasing these optimizations vs. negativity in the press it’d be an easy decision (not to mention the whole morality argument),” Lal Shimpi and Klug report. ” It’s also worth pointing out that nearly almost all Android OEMs are complicit in creating this mess. We singled out Samsung for the initial investigation as they were doing something unique on the GPU front that didn’t apply to everyone else, but the CPU story (as we mentioned back in July) is a widespread problem.”

Read more in the full article here.

MacDailyNews Take: Have fun trying to fake a 64-bit OS on a 64-bit chip, sleazebags.

[Thanks to MacDailyNews Reader “macnut222” for the heads up.]

Related articles:
Apple slams Samsung’s serial benchmark deceptions – October 2, 2013
Samsung shenanigans inflate Note 3’s benchmarking scores by up to 20% – October 1, 2013
Samsung caught doping Galaxy S 4 benchmarks – July 30, 2013

27 Comments

  1. This needs much wider press coverage. This is an actual Antenagate and the are caught red handed.

    I’m sure the media will do the right thing (heaviest sarcasm ever) and let everyone know.

    1. “I’m sure the media will do the right thing (heaviest sarcasm ever) and let everyone know.”

      This IS the media. MDN, like so many other independent news gatherers/editors (Drudge, et al) are on the brink of replacing CiaNN, CBS Evening Lies, The Washington Compost, etc..astute people are sick of being lied to. In the new media, every person with an iPhone can be a journalist.

  2. Benchmarking for phones is only useful when comparing commodity products that are more or less indistinguishable. So if most Android phone makers are doing this cheating, I guess the playing field is level. The player who cheats best wins the game.

    However, when you put iPhone into consideration, benchmarking becomes irrelevant. Who made the decision to buy an iPhone because of benchmarking results? No one… 🙂

    1. Indeed. The Android lovers absolutely fall over themselves to talk specs and benchmarks, but I’ve yet to meet a single iPhone owner care about those beyond saying that their new iPhone *insert model* is noticeably faster than the iPhone *insert model* they had before. iPhone owners mostly bang on about the cool little tricks in the OS they’ve just noticed, not some abstract number that nobody gives a toss about.

      1. Excellent point: Teaching To The Test indeed. Crap in, crap out in both cases.

        There has to be a huge book concept in there somewhere. The demented philosophies that drove a brilliant civilization to literally destroy itself from within. Biznizz bozoid behavior is only one small aspect of the self-destruction. Maybe call it the concept of self-cannibalization? Eat or be eaten? Dog eat dog?

  3. Seems like a class action lawsuit should be implemented. How many people chose Android over alternatives based on specifications? I have a friend who I told to get an iPhone, he said he chose an Android phone because he was impressed by its specs compared to the iPhone. How many suckers, err, I mean customers have done the same? Can you imagine the pundits if Apple had done something like this? We’d see an immediate 10% drop in Apple’s stock value and people screaming for their head.

  4. The problem with bad character is it is uncorrectable. They will continue to do this and must be financially punished every time until they are gone. Sort of like those who would sit home and receive free benefits from those who work for a living.

    1. Ahh but you miss the cultural karma that accumulates to a Korean company that ‘pulls strings’.

      This is considered what an executive does to show he is a good soldier in the company and will not stop at anything reasonable to do to benefit his masters.

      1. That doesn’t play as well outside of South Korea itself. Yet the impulse to behave as you describe is persistent and may damage them in foreign markets. Already the buzz over their supposed innovation is shriveling. There are signs they may move beyond Android, a dead end in the quicksand in the fog, to a platform of their own. That will take time, and the internal corporate culture you refer to may not favour its adoption in time to save their market share from decline.

    1. Interestingly, Qman, the Korean automakers are getting flak on the automotive forums for doing just that. They have no shame, and fear no courts. They will do anything it takes legal or otherwise to give themselves an advantage, and they know that being dragged into the courts in the US is a win for them anyway, as after multiple appeals, so much time elapses, that the issue is no longer valid for new models, and the plaintiffs tire and settle. They end up making more money than their legals fees, so they don’t care if it’s legal or not.

    2. yep, Hyundai has been caught doing exactly that. They had to restate their fuel economy claims for the Elantra, but not before clusterbombing the airwaves and websites with ads touting its “40 MPG” claims. The corrections eventually made their way onto the window stickers, but obviously not every source was updated to reflect the corrected numbers. Hyundai claimed that it was an honest mistake during the testing. Mmmm hmmm.

  5. I really don’t understand this at all. Knock-offs are for people who can’t afford the real thing. They’re already resigned to having shit performance. Why does a company like Samsung bother to cook the books, knowing how much damage their reputation will take when they’re caught?

    -jcr

    1. Because their reputation does not get damaged when they get caught. How many reports have you seen about Samsung’s roided up benchmark tests versus “Antennagate”?

      And Samsung has insulated itself from a lot of this stuff with their huge marketing outlays (far higher than Apple’s marketing budget), paid-off bloggers and commenters, and other astroturfing campaigns. A story like this is simply not going to gain traction, especially since now the story is starting to shift towards other Android OEMs that do the same thing. Samsung can simply claim that “everybody does it” and the media will stop right there.

      1. Agreed, they are not too worried. Their revenue streams are far less dependent on reputation than a brand like Apple’s. The basis of competition is different—they observe what is selling and make their own version of it, an efficiency that bypasses expensive R&D ramp-up.

        The only problem with this inspired approach is court challenges over intellectual property rights. But as long as courts rule from a basis of technological idiocy, or the appeals process lasts longer than a product cycle, Samsung will make money faster than any fines can be accrued.

        One might pray for a single enlightened judge who sees through the charade and imposes punitive damages. Prayer, however, like common sense, does not have a good track record in any venue, and thus in any venue, wars must continue.

        What a sorry excuse for a sentient species this is, as my sainted mother mournfully intoned. More than once.

  6. Benchmarking, I say WGAS. But, if any of these geeks wanted to impart some actual value to the process, why not compare performance on some popular, cross-platform apps. That way, if a mfgr wanted to “cheat,” we’d realize some benefit beyond looking at each other in the locker room.

  7. MDN deletes posts that speak adversely against Apple giving the impression that all posters here are Apple zombies drinking the Kool Aid. So what’s the difference between fucking MDN & Samsung? Not a whit.

  8. I don’t agree. It’s too easy to get away with so that’s why they do it. The benefits outweighs the negative none existent PR. Until now there has been no mention of this.

    This reminds me of the bastard company Nvidia that many years ago tweaked their drivers to perform better in Benchmark programs on the PC… They got busted.

Reader Feedback

This site uses Akismet to reduce spam. Learn how your comment data is processed.