Dopey ‘study’ conjures up Apple iPhone typing ‘errors’

In a November 13, 2007 press release headlined, “Direct Comparison of iPhone and Hard-Key QWERTY Phone Owners Indicates Higher Text Entry Error Rate for iPhones,” User Centric, Inc., a Chicago-based usability consultancy, states that they have finished a third and final study examining the user experience of Apple’s iPhone.

Our study involved data from 60 participants who were asked to enter specific text messages and complete several mobile device tasks. Twenty of these participants were iPhone owners who owned their phones for at least one month. Twenty more participants were owners of traditional hard-key QWERTY phones and another twenty were owners of numeric phones who used the “multi-tap” method of text entry.

MacDailyNews Take: Everything you are about to read from this point on from User Centric’s press release (in italics) is virtually meaningless and applies only to a miniscule sample of 20 people who have owned an iPhone for at least a month plus 40 other non-iPhone users. The likelihood of the sample size being too small to be useful to anyone outside this small group of 60 people is extremely high. We could pick another random 20 iPhone users and get completely different results. Were these 20 iPhone owners selected because they were heavy users of text messaging or did they get their iPhones primarily so they could browse the real Web or perhaps so they could have a multi-touch iPod or maybe they use email over SMS texting? How many, if any, of these iPhone users even use their iPhone regularly or at all for texting? What were the ages of these 20 iPhone owners? Was it a group of 35-year-old soccer moms who don’t know SMS from PMS or were they 16-year-olds who text in their sleep? By the way, how long were their fingernails? The questions are endless. Without such salient details being disclosed, the tiny sample size of 20 people becomes even more of an issue. This “study” is fatally flawed.

Participants were brought in for 75 minute one-on-one usability sessions with a moderator. Each participant entered six fixed-length text messages on their own phone. Non-iPhone owners also did six messages each on the iPhone and a phone of the “opposite” type. The opposite phone for numeric phone owners was a Blackberry and for hard-key QWERTY phone owners it was a numeric Samsung E300 phone.

MacDailyNews Take: Oh, so iPhone users got to enter 6 messages on their iPhones, but not the other phones? However, the other 40 people who had never touched an iPhone got to enter 6 messages on the iPhones? Why? To inflate the number of “mistakes” made on the iPhones?

When compared to hard-key QWERTY phone owners using their personal phones, iPhone owners’ rate of text entry on the iPhone was equally rapid. However, iPhone owners made more errors during text entry and also left significantly more errors in the completed messages.

MacDailyNews Take: The full press release details the meaningless stats that this tiny, undefined sample of just 20 iPhone users generated.

Furthermore, when iPhone owners were asked to perform a text correction task during their sessions, 21% of iPhone owners were not aware of the magnifying glass correction feature although they had owned their iPhone for one month. Participants who did know about the feature clearly loved it, and participants who were new to it indicated that it would be useful in the future.

MacDailyNews Take: First of all, how do you get “21%” out of a sample of 20 iPhone owners? 4.2 of the iPhone owners didn’t know about the magnifying glass feature? What happened to the other 0.8 of that individual iPhone user? Maybe he thought he knew about it, but forgot? So, the sample size is too small and too undefined and now it’s flawed to boot: Tell all of the participants about the tools available or your study’s results are junk. The fact is that the magnifying glass tool is available and all users should have been informed and allowed to use it. This would have allowed the survey to gather meaningful results if, of course, they had used a reasonable sample size, which they didn’t. Did they also neglect to tell 21% of the hard-key keyboard users where the arrow keys were or how to use them?

Participants who had previously not used either a hard-key QWERTY phone or an iPhone were significantly faster at entering text messages on the hard-key QWERTY test phone than on the iPhone. These participants also made significantly fewer errors on the hard-key QWERTY than on the iPhone.

MacDailyNews Take: So, had these participants used a hard-key QWERTY keyboard on something else like, say, a personal computer? Of course they had, and likely for many, many years. Certainly most people have vastly more experience with hard-key keyboards vs. users of iPhone keyboards with “at least a month” of experience. That experience gap would be accounted for in a real study. It is ignored here. All of the participants are naturally inclined to be more familiar with a hard QWERTY keyboard whether it be on a phone or not. It’s only logical. But, the last thing those who conducted this “study” want you to employ is logic; either that or they’re just incompetent.

Numeric phone owners made an average of 5.4 errors/message on the iPhone, 1.2 errors/message on the QWERTY test phone, and 1.4 errors/message on their own phone. “Not only was their performance better,” says Jen Allen, User Experience Specialist for User Centric, “their rankings and ratings of the phones indicated that they preferred a hard-key QWERTY phone for texting.”

MacDailyNews Take: Oh, so those familiar with QWERTY keyboards and wholly unfamiliar with iPhone’s keyboard performed better on a QWERTY hard-key keyboard and ranked it higher? Stunning finding there, Jen. One would also guess that there’s an 1879 “study” in a dusty file cabinet somewhere that found people preferred candles to electric light bulbs for reading. No doubt commissioned by the wax industry.

User Centric also conducted a “detailed analysis of text entry patterns” on 20 random people which came up with a bunch more meaningless numbers which are used to either obfuscate the appalling ridiculousness of the study’s useless sample size and methodology or, again, they’re just so incompetent that they think the numbers have meaning.

Compared to hard-key QWERTY devices, the iPhone may fall short for consumers who use on their mobile device heavily for email and text messaging.

MacDailyNews Take: Or iPhone may not fall short. The iPhone may be a heavy email and text messaging user’s godsend. We simply don’t know either way from this study. Among many other faults, the sample size is too small and too undefined and they didn’t let the iPhone users touch the other phones.

The iPhone was clearly associated with higher text entry error rates than a hard-key QWERTY phone.

MacDailyNews Take: Only in this particular “study” with its tiny sample size and flawed methodology: Letting non-iPhone users contribute mistakes to iPhones while keeping iPhone users off the other phones, thereby logically inflating the non-iPhones’ accuracy. Hard-key users, regardless of type of phone keyboard would naturally be more familiar with hard-key keyboards and therefore make fewer mistakes in use.

“The iPhone is a great switch from a numeric phone. But if you’re switching from a hard-key QWERTY phone, try the iPhone in the store first,” recommended Gavin Lew, User Centric’s Managing Director.

MacDailyNews Take: More garbage, which is a fitting end. Trying an iPhone in a store sounds like a great idea, but you need to use it for at least a few days to learn how it works and get used to it (and it to you). If you base your decision on how you typed on an iPhone in a store for a few minutes, most likely your conclusion will be exactly the wrong one – which conveniently seems to support the findings of this “study.” It’s no wonder Lew wants buyers to base their opinion of iPhone’s keyboard on laughably insufficient testing, as that’s exactly what his firm’s “study” does.

We encourage readers to read the full press release while employing logic and paying close attention to which users were allowed to use which device(s), how few people were involved, and what exactly was tested, here.

We also encourage readers to note any media outlets that dutifully report the “findings” of this “study” as fact.

MacDailyNews Note: On their website, User Centric lists their “clients over the last two quarters” which include: Verizon Wireless, Microsoft, LG, and Motorola.

64 Comments

  1. Actually this is one of the few functions of the iPhone that are an issue for me. Being a fanatic so to speak I love the iPhone of course but typing while driving sure can be a problem. I know I shouldn’t!

    Also please come out with a copy and paste function!!!!!!

    Oh yeah and a function to add a contact from the recent calls list that automatically carries the phone number over!

  2. Direct comparisons of eating iPhone glass and eating ice cream show clearly that the iPhone sucks.

    For our next direct comparison, we will compare iPhone’s smooth surface to the Himalayas and Jessica Simpson’s face before Proactiv Solution.

    How about a direct comparison between other all touch screen surfaces? Ugh.

  3. The premise of this study is obvious to test the iPhone, not the other phones.

    I could call it a real study if EACH user typed that same message on a different phone each day during the study (to prevent thumbs from getting tired).

    THEN if they said the iPhone produced significant errors, I could believe it. As it is, this study is just plain BAKA! (Japanese: stupid)

  4. I’m only a sample of one, but I love the touch keyboard – I furiously type without regard for accuracy – the predictive correction works wonderfully for me except for proper names (duh), where I have to slow down and eyeball the word. The newest feature of double spacing adding a period makes 99% of texting very easy too

  5. Note also that iPhone users had their phones for “at least one month” (most may have been relatively new at it) while the hard-key phone users had their phones for an unstated period of time (very likely a much longer time.) So they may well be testing people with a ton of experience on hard-key phones against people who never used a phone keyboard of any kind before they bought an iPhone. We just can’t tell from the information provided. As a former ad agency guy who has witnessed decades of bogus testing masquerading as legitimate research, I can tell you — I smell something extremely fishy here.

  6. It really suck to read and article when Mac Daily News is stuffing their two cents worth in after every sentence. By the way, I didn’t read your take because we all know you are way to far up apple’s ass to accept that any apple product may have a shortcoming.

    Let us read the article first then make your arguments.

  7. “Oh yeah and a function to add a contact from the recent calls list that automatically carries the phone number over!”

    You can do this on the iPhone. In your Recent Calls list, press the blue arrow beside the phone number, then press the ‘Create New Contact’ or ‘Add to Existing Contact’ as suits your fancy. Either way, the phone number automatically carries over, as you would expect.

Reader Feedback

This site uses Akismet to reduce spam. Learn how your comment data is processed.