“Macs tend to fare second best in Consumer Reports testing, partly because the magazine lives in ignorance of the differences between Apple’s computers and Windows boxes,” Gene Steinberg writes for The Tech Night Owl. “But they’ve always been recommended, until recently. I can quibble about the way the tests appear to emphasize features over performance, usability and reliability. In fact, I have.”
“It all started when CR reported wildly divergent battery life results, ranging from 3.75 hours up to 19 hours over three tests for each product. The latter is way more than Apple’s estimates, which range up to 10 hours,” Steinberg writes. “The inconsistency didn’t make sense, and thus marketing VP Philip Schiller posted a tweet — the new normal for getting the word out nowadays — saying that the results didn’t jibe with Apple’s own field tests.”
“Now CR’s tests are intended to be consistent from notebook to notebook. It involves downloading 10 sites from the company’s in-house server until the battery is spent. So just what was going on here, and was the test deliberately designed to leave Safari — and Macs — second best?” Steinberg writes. “Well, that’s debatable, but to achieve consistent results, CR turns off caching on a browser. With caching on, the theory goes that the sites would be retrieved from the local cache, which presents an anomalous situation since different computers — and operating systems — might do it differently. On the other hand, it would also be using the computer normally, not in an artificial way.”
“How can such a test possibly produce results that in any way reflect what a typical user would encounter?” Steinberg writes. “CR should have realized something was amiss as soon as the battery life normalized with caching on. They could have reached out to Apple before the results were published for clarification. As it was, CR got a boatload of publicity for its decision not to recommend the MacBook Pros. Of course, that result will soon be changed if all goes well.”
Read more in the full article here.
MacDailyNews Take: Consumer Reports is, was, and has always been a joke when it comes to testing anything remotely associated with tech to say nothing of devices running the world’s most advanced operating systems. They don’t get the Mac (or the iPhone or the iPad) and they never will.
As Hanlon’s Razor states: Never attribute to malice that which is adequately explained by stupidity.
Consumer Reports is an anachronism for grandma to use to reassure herself that she bought the right vacuum cleaner (even though she didn’t unless she bought a Miele – which she almost certainly didn’t since she’s a Consumer Reports subscriber).
SEE ALSO:
Consumer Reports’ weird MacBook Pro battery test results due to use of obscure Safari developer setting – January 10, 2017
Consumer Reports stands by its weird MacBook Pro battery test results – December 29, 2016
Consumer Reports says do not buy Apple’s new MacBook Pro, citing erratic battery life – December 23, 2016
Consumer Reports evaluates iTunes Store movie streaming, confusion ensues – August 13, 2012
Is Consumer Reports having its revenge against Apple? – July 10, 2012
How Apple and facts killed Consumer Reports – March 29, 2012
Consumer Reports was no iPhone killer and they’re no iPad killer, either – March 28, 2012
Tests prove Apple’s new iPad heat levels comparable to Android tablets – March 26, 2012
Expert: iPad heat claims overblown, not a real issue – March 22, 2012
What’s the deal with Consumer Reports and Apple? – March 21, 2012
Consumer Reports’ bombshell: New iPad runs hotter than predecessor but ‘not especially uncomfortable’ – March 20, 2012
FUD Alert: Consumer Reports to ‘investigate’ reports of iPad and ‘excess heat’ – March 20, 2012
Consumer Reports hops off free PR gravy train, officially recommends Apple iPhone 4S – November 8, 2011
Why does anyone believe Consumer Reports? – April 6, 2011
Consumer Reports on iPad 2: We didn’t notice any significant speed improvement – March 15, 2011
Consumer Reports was wrong on Verizon iPhone 4; so-called ‘death grip’ fixed by Apple – March 2, 2011
Consumer Reports: Verizon iPhone 4 has antenna ‘problem’; not recommended – February 25, 2011
Consumer Reports continues laughable vendetta against iPhone 4 – January 14, 2011
Android sweeps Consumer Reports’ rankings as iPhone 4 is omitted – November 17, 2010
All of Consumer Reports’ ‘recommended’ smartphones suffer attenuation when held – July 19, 2010
Consumer Reports: Apple’s free Bumper case does not earn iPhone 4 our recommendation – July 16, 2010
Consumer Reports: Apple’s Bumper case fixes iPhone 4 signal-loss issue – July 15, 2010
Consumer Reports continues harping on iPhone 4 attenuation issue – July 14, 2010
Electromagnetic engineer: Consumer Reports’ iPhone 4 study flawed – July 13, 2010
The Consumer Reports – Apple iPhone 4 fiasco – July 13, 2010
Consumer Reports: Oh yeah, almost forgot, Apple iPhone 4 is also the best smartphone on the market – July 12, 2010
Consumer Reports: We cannot recommend Apple iPhone 4 – July 12, 2010
Consumer Reports does their readership a disservice, says viruses target Apple Macs – December 13, 2005
Consumer Reports: Apple’s new iPod screens scratch-prone like iPod nanos – October 28, 2005
Consumer Reports dubiously finds 20-percent of Mac users ‘detected’ virus in last two years -UPDATED – August 10, 2005
Do you laugh or cry here? So you don’t judge a product other than through its own pre determined way of doing things which by definition can lead to anything but a fair comparison and internally created skewing of the truth to suit its own prejudices… or worse as the take alludes to. What possible sense is there to take away the advantages any product offers in everyday use to create a phone result that celebrates inadequacies by others? Just imagine what they could get up to in car comparisons using that logic, a Porsche would be no faster than a Trabant if that is the way they wanted to present it.
Not so fast MDN and fanboys. This does not explain:
-Why Apple took such a drastic step and ELIMINATED TIME LEFT estimates in OS X;
-Why multiple reviewers, including The Verge, stated that the battery life on these new MacBook Pros is less than what Apple advertises and is inconsistent.
There are also reviews which praise the battery life. ex. Trustedreviews.com, etc. For that matter, as someone who now owns a 15″ model, I can tell you first hand that the battery life is great. I have no trouble going all day on the battery and see results right in line with Apple’s estimates of about 10 hours.
As someone who has a technical mind dswe, you likely realize that users do not use the power of their computers at a consistent rate. When a user uses power at a low rate to browse the web or read email, the time left would be calculated based on that low consumption. When they start a heavy demand task, the power use shoots up and the time recalculation would show very little time left at that rate of consumption. This varying time has lead some users to think that the first (long) time was applicable for the entire computer usage and when they are doing some heavy work, the time seems to be almost gone. Many people panic at a situation like that and run for the charger and then complain that the battery doesn’t last. It is a calculus thing. Predicting the length of time remaining may not be linear and confuse the uninitiated.
I vote for stupidity.
CR should have reported that their testing procedures don’t involve typical user conditions. Furthermore, I believe they should test computers as designed by their manufacturers to operate. That’s how the VAST majority of their readers would use them.
If they want to test under both normal and artificial but very consistent conditions, that’s OK, but they need to explain to readers what they were doing. And if they had done that and noticed wildly divergent results, I would like to believe they would have reached out to Apple to understand why, so as to explain to their readers the reason and state that the divergent results would be unlikely to be experienced by most users. That would be honest reporting.
My assumption is that they didn’t know why they got such weird results, that no one else gotten, but to consider that it was somehow related to their rather unique testing procedures never occurred to them, beyond perhaps checking their equipment. To consider that it was somehow related to the procedures they use to test was unthinkable.
A future correction should explain all this. We shall see if that, indeed, does happen. If it does, I hope MDN will link to that report.
I hope sites STOP linking to CR. That is what CR wanted – attention! They have largely become irrelevant due to the vast availability of reviews from the internet. IMO, the ONLY thing that they had going was ‘objectivity’ … but they blew that a long time ago. Not only Apple with ‘antenna-gate’ but other clearly biased and non objective (non Apple) reviews.
“And if they had done that and noticed wildly divergent results, I would like to believe they would have reached out to Apple to understand why … That would be honest reporting.”
Very true. But CR has been biased against Apple for a very long time. I might trust them when I buy a toaster, but I would never trust them about buying a Mac. They’ve got an agenda.
Apple earned the CR endorsement for every previous Macbook Pro. I see no bias at all
What happened here is Apple failed an objective test that was run consistently on other laptops including previous Macs. Apple failed because of shoddy software, their own admission. I say Apple also fails to provide pro users a good value when the battery is so inadequate. The ssd and ram are soldered in. This is an executive fashion accessory with the misused Pro name. Stop bitching about CR when it’s Apple that fucked up. Again.
I simply consider Consumer Reports unprofessional and unscientific. As such, I don’t know what they’re good for at this point. 🙃
The haterade against CR is ridiculous. If you have a more fair test to run, please post your process. The battery life test CR ran is a good one. The goal of a characterization test is to be consistent, and comparable to other device tests, not to simulate YOUR specific use case.
ARS Technica has an excellent report on this:
http://arstechnica.com/apple/2017/01/consumer-reports-will-re-test-macbook-pro-batteries-after-apple-fixes-bugs/
“To test battery life, Consumer Reports sets laptop screens to a specific brightness level and then loads a series of webpages in the laptop’s default browser (Safari in this case) in a loop until the battery dies. Apple suggests that disabling browser caching for a test like this doesn’t reflect real-world use, but it does make sense for a synthetic test—users will continually read new pages rather than visiting the same static pages over and over again, so Consumer Reports wants to make sure that its test is actually downloading data over the network rather than simply reading cached data from the disk.”
It is also well reported that Apple chose an inferior battery for the 2016 MBP line in order to meet Ive’s thinnness requirement rather than offering truly robust battery performance. CR isn’t at fault for stupid design decisions on Apple’s part.
I lost interest in the first paragraph. Any trained scientist could have designed a fair and meaningful test and stated rational results. CR is apparently INCAPABLE of such a test.
No, I won’t bother writing up a test for CR. My must be insane today. I skipped the rest. Don’t care.
Correction: YOU must be insane today.
How many “Mike”s do we have around here?
Derek, I thought you were more intellectually honest than that. Understand the facts before accusing the messenger. CR did a good job, Apple did not. That is the truth.
Intellectually honest when you haven’t read everything I’ve written on the subject before bombing me with crazy stuff.
I’ve consistently made it clear that when Apple screws up, I want to know, and I will bitch about it. The new MBP battery situation is something I have ALREADY addressed as such.
As for Consumer Reports, you clearly have no clue how a scientific study is defined, planned, carried out and verified. Go learn about it and leave people who DO know about it alone, free of your ignorance. I fully stand by exactly what I’ve said about Consumer Reports.
I will also say that the deep damage perpetrated by Consumer Reports by their blatant and willful INCOMPETENCE should rightfully kill off their customer base and throw them in to bankruptcy. Pulling this level of INCOMPETENCE while calling it ‘professional’ is DANGEROUS.
Let’s get some competent people performing work at CR, or let’s see CR go bankrupt to be succeeded by someone dedicated to professional and scientific practices.
Are you happy now that I’ve further bitched on the matter? You inspired it.
I don’t believe you have addressed Apple’s screwup when it comes to MBP battery performance. You and the usual fanboys are just trashing CR because their objective test — a test run by all other laptops — did not yield the results that Apple claimed. Do please point to where you have designed a more scientifically valid test.
You are implying is that CR test MUST be incorrect because it doesn’t match Apple’s MARKETING pitch. That is fanboyism at its worst. Well now CR has rerun its tests with Apple’s software patches installed and voila, battery life is indeed improved. Problem was squarely on Apple, not on CR. You just didn’t like the fact that an objective test agency published what you didn’t want to hear.
Before the patch, some users in the real world are getting less than 4 hours of battery life from their new MBPs in real world usage. So they must be doing everything wrong, I guess. Apple is perfect, right? Well we would never know because Apple refuses to publish its test method.
Apple could have avoided the whole issue by putting in a properly sized battery and writing bug-free software. Call Apple on it if you have a greater spine than many of the fanboys here. The juvenile attacks on CR are unwarranted.
We’ve been over all this. 💤
Mike, CR’s testing methodology doesn’t reflect real world usage in any way shape or form. If you have to put a product in developer mode and effectively hack the device to do things like disabling caches, etc., then your test becomes very synthetic by nature and not representative of real world conditions. Worse, CR actually believes they are “simulating” real world conditions.
A bigger issue I have with CR in this particular case is that responsible reporting would have had them contact the vendor to help explain the inconsistent and anomalous results they were getting before they want to publish their report. As it stands now, their flawed testing methods have been exposed as well as a lack of competence on their part.
Moreover, CR’s incompetence isn’t unique to Apple or even computers. Historically, their reviews of photography equipment as been equally bad. They’re probably fine for evaluating toaster ovens, but consumer electronics is clearly out of their range of competence.
Having just read the ARS spin on this, I didn’t see anything in the article which is new information not covered in CR’s own response. Likewise, I’m not sure what it is about their report that you consider “excellent”. In fact, they simply give CR a pass on their flawed testing methodology and blame CR’s mistakes on a bug in Safari. Yes, there was a bug in Safari, but not a bug that anyone other than a developer would ever see.
To Ars credit, they did mention the point I made in my post above…
” Apple suggests that disabling browser caching for a test like this doesn’t reflect real-world use, but it does make sense for a synthetic test”.
Apple suggests? Really? Sorry, but that’s just common sense.
I have actually done benchmarking professionally for hard drives. CR started with the right idea: reduce variability so that you can actually compare different hardware with each other. There are many sources of variability and not all of them can be contained. (One problem we had was with the OS itself: a single write could take 10x longer than the next one due to uncontrollable factors.)
Arguably, CR screwed up firstly by using a benchmark that did not accurately reflect the actual use by consumers. But again, “actual use by consumers” adds uncontrollable random factors, so you have to approximate.
The big problem with CR is that when they recorded results that were unexpected (different from what they should have been or different from one run to another), they should have stepped back and reevaluated their processes. Again, this is a judgement call: it would delay their publication of results and so deny buyers important information that they need.
Given the gross inconsistency of the result, they should have delayed publication to resolve it.
Well, the fanbois claim CR is biased against Macs because…they have Macs run the SAME TEST that PCs run?
It’s not so much a bias against Macs as it is an issue of incompetence. Given the wild swings in their results (anywhere from 3 to 18 hours of battery life), a reputable publication would have dug a bit deeper into the issue a even contacted the vendor if they didn’t have the technical capabilities before publishing their meaningless results. Through this process, Apple doesn’t come out of this looking bad… CR does.
There cannot be an even up comparison because Safari does not allow turning off caching other than entering developer mode. What else does developer mode do?
Would running Chome across platforms give a better comparison? I don’t know…
What we (correctly) can’t have is comparing cached browsers versus uncached. Uncached is the fair test, because its battery life, not cache efficiency being measured.
What CR should do is load a thousands of different web pages to load, dense in data, and everyone different, enough to never have a page load from Cache. Then they leave the settings for every computer normal user settings and test away. Or have an 1/2 ton of pages and purge the cache periodically – a set interval the same for every computer.CR needs scientifically trained test dsigners, and needs to avoid shortcuts that distort results.
Putin hacked CR’s database.
CR may not be a site for computer performance reference but before they published their results there were many independent users reporting battery inconsistencies, and that is why Apple removed the Time left indicator claiming “it wasn’t accurate enough”. Besides it is rational to at least consider a premise before coming to a conclusion, in this case it wasn’t a conclusive review.
I remember a few weeks ago a user here in MDN claiming less than 4 hours on his new MBP, so is this “inconsistency problem” over?
Your comments are a joke.
There was a bug that caused the apple mac books to consume a lot of power via Safari – Apple just fixed it. That you didn’t experience it doesn’t mean you wouldn’t have- depends on what you were doing.
It’s fixed now. That’s the whole point of CR’s testing they have a lot of respect among manufacturers. You guys are big babies when your “Apple” has some bites taken out of it.