In a November 13, 2007 press release headlined, “Direct Comparison of iPhone and Hard-Key QWERTY Phone Owners Indicates Higher Text Entry Error Rate for iPhones,” User Centric, Inc., a Chicago-based usability consultancy, states that they have finished a third and final study examining the user experience of Apple’s iPhone.
Our study involved data from 60 participants who were asked to enter specific text messages and complete several mobile device tasks. Twenty of these participants were iPhone owners who owned their phones for at least one month. Twenty more participants were owners of traditional hard-key QWERTY phones and another twenty were owners of numeric phones who used the “multi-tap” method of text entry.
MacDailyNews Take: Everything you are about to read from this point on from User Centric’s press release (in italics) is virtually meaningless and applies only to a miniscule sample of 20 people who have owned an iPhone for at least a month plus 40 other non-iPhone users. The likelihood of the sample size being too small to be useful to anyone outside this small group of 60 people is extremely high. We could pick another random 20 iPhone users and get completely different results. Were these 20 iPhone owners selected because they were heavy users of text messaging or did they get their iPhones primarily so they could browse the real Web or perhaps so they could have a multi-touch iPod or maybe they use email over SMS texting? How many, if any, of these iPhone users even use their iPhone regularly or at all for texting? What were the ages of these 20 iPhone owners? Was it a group of 35-year-old soccer moms who don’t know SMS from PMS or were they 16-year-olds who text in their sleep? By the way, how long were their fingernails? The questions are endless. Without such salient details being disclosed, the tiny sample size of 20 people becomes even more of an issue. This “study” is fatally flawed.
Participants were brought in for 75 minute one-on-one usability sessions with a moderator. Each participant entered six fixed-length text messages on their own phone. Non-iPhone owners also did six messages each on the iPhone and a phone of the “opposite” type. The opposite phone for numeric phone owners was a Blackberry and for hard-key QWERTY phone owners it was a numeric Samsung E300 phone.
MacDailyNews Take: Oh, so iPhone users got to enter 6 messages on their iPhones, but not the other phones? However, the other 40 people who had never touched an iPhone got to enter 6 messages on the iPhones? Why? To inflate the number of “mistakes” made on the iPhones?
When compared to hard-key QWERTY phone owners using their personal phones, iPhone owners’ rate of text entry on the iPhone was equally rapid. However, iPhone owners made more errors during text entry and also left significantly more errors in the completed messages.
MacDailyNews Take: The full press release details the meaningless stats that this tiny, undefined sample of just 20 iPhone users generated.
Furthermore, when iPhone owners were asked to perform a text correction task during their sessions, 21% of iPhone owners were not aware of the magnifying glass correction feature although they had owned their iPhone for one month. Participants who did know about the feature clearly loved it, and participants who were new to it indicated that it would be useful in the future.
MacDailyNews Take: First of all, how do you get “21%” out of a sample of 20 iPhone owners? 4.2 of the iPhone owners didn’t know about the magnifying glass feature? What happened to the other 0.8 of that individual iPhone user? Maybe he thought he knew about it, but forgot? So, the sample size is too small and too undefined and now it’s flawed to boot: Tell all of the participants about the tools available or your study’s results are junk. The fact is that the magnifying glass tool is available and all users should have been informed and allowed to use it. This would have allowed the survey to gather meaningful results if, of course, they had used a reasonable sample size, which they didn’t. Did they also neglect to tell 21% of the hard-key keyboard users where the arrow keys were or how to use them?
Participants who had previously not used either a hard-key QWERTY phone or an iPhone were significantly faster at entering text messages on the hard-key QWERTY test phone than on the iPhone. These participants also made significantly fewer errors on the hard-key QWERTY than on the iPhone.
MacDailyNews Take: So, had these participants used a hard-key QWERTY keyboard on something else like, say, a personal computer? Of course they had, and likely for many, many years. Certainly most people have vastly more experience with hard-key keyboards vs. users of iPhone keyboards with “at least a month” of experience. That experience gap would be accounted for in a real study. It is ignored here. All of the participants are naturally inclined to be more familiar with a hard QWERTY keyboard whether it be on a phone or not. It’s only logical. But, the last thing those who conducted this “study” want you to employ is logic; either that or they’re just incompetent.
Numeric phone owners made an average of 5.4 errors/message on the iPhone, 1.2 errors/message on the QWERTY test phone, and 1.4 errors/message on their own phone. “Not only was their performance better,” says Jen Allen, User Experience Specialist for User Centric, “their rankings and ratings of the phones indicated that they preferred a hard-key QWERTY phone for texting.”
MacDailyNews Take: Oh, so those familiar with QWERTY keyboards and wholly unfamiliar with iPhone’s keyboard performed better on a QWERTY hard-key keyboard and ranked it higher? Stunning finding there, Jen. One would also guess that there’s an 1879 “study” in a dusty file cabinet somewhere that found people preferred candles to electric light bulbs for reading. No doubt commissioned by the wax industry.
User Centric also conducted a “detailed analysis of text entry patterns” on 20 random people which came up with a bunch more meaningless numbers which are used to either obfuscate the appalling ridiculousness of the study’s useless sample size and methodology or, again, they’re just so incompetent that they think the numbers have meaning.
Compared to hard-key QWERTY devices, the iPhone may fall short for consumers who use on their mobile device heavily for email and text messaging.
MacDailyNews Take: Or iPhone may not fall short. The iPhone may be a heavy email and text messaging user’s godsend. We simply don’t know either way from this study. Among many other faults, the sample size is too small and too undefined and they didn’t let the iPhone users touch the other phones.
The iPhone was clearly associated with higher text entry error rates than a hard-key QWERTY phone.
MacDailyNews Take: Only in this particular “study” with its tiny sample size and flawed methodology: Letting non-iPhone users contribute mistakes to iPhones while keeping iPhone users off the other phones, thereby logically inflating the non-iPhones’ accuracy. Hard-key users, regardless of type of phone keyboard would naturally be more familiar with hard-key keyboards and therefore make fewer mistakes in use.
“The iPhone is a great switch from a numeric phone. But if you’re switching from a hard-key QWERTY phone, try the iPhone in the store first,” recommended Gavin Lew, User Centric’s Managing Director.
MacDailyNews Take: More garbage, which is a fitting end. Trying an iPhone in a store sounds like a great idea, but you need to use it for at least a few days to learn how it works and get used to it (and it to you). If you base your decision on how you typed on an iPhone in a store for a few minutes, most likely your conclusion will be exactly the wrong one – which conveniently seems to support the findings of this “study.” It’s no wonder Lew wants buyers to base their opinion of iPhone’s keyboard on laughably insufficient testing, as that’s exactly what his firm’s “study” does.
We encourage readers to read the full press release while employing logic and paying close attention to which users were allowed to use which device(s), how few people were involved, and what exactly was tested, here.
We also encourage readers to note any media outlets that dutifully report the “findings” of this “study” as fact.
MacDailyNews Note: On their website, User Centric lists their “clients over the last two quarters” which include: Verizon Wireless, Microsoft, LG, and Motorola.