“A new report about Apple Inc.’s voice-recognition software Siri concludes what many users have been saying for a while: It doesn’t work all that well,” Adam Satariano reports for Bloomberg.
“Of 1,600 common searches, the speech technology accurately resolved the request 62 percent of the time on a loud street and 68 percent in a quieter setting, according to a report released today by Piper Jaffray Cos., the Minneapolis-based investment bank,” Satariano reports. “The report graded the technology ‘D’ for accuracy, while predicting it will improve as more features are added. ‘You’re playing the lottery when you’re using Siri,’ said Gene Munster, the Piper Jaffray analyst who conducted the study. ‘They have a plan to be more competitive, but it’s going to take a couple of years.'”
MacDailyNews Take: Siri is a beta and it is currently severely overtaxed, according to whispers we’ve heard. “Siri, what do you need?” Siri: “More data centers, please!”
Satariano reports, “Munster said that while Siri is good at comprehending what a user is saying and will accurately repeat the question, it struggles turning those words into a correct answer. For instance, Siri will repeat old answers when a user is trying to ask a new question. The technology also struggles when trying to use speech commands to find directions, Munster said. In Piper Jaffray’s tests, Siri was able to accurately decipher what a user was saying 83 percent of the time on the street and 89 percent in an area with low noise. ‘Apple right now gets a ‘B’ in comprehension and a ‘D’ in accuracy,” Munster said. ‘There’s a big difference between comprehension and her actually doing what you want her to do.'”
Read more in the full article here.
MacDailyNews Take: Siri users, what letter grades would you give to Siri (queries, not dictation, which gets an “A-” in our grade book) for comprehension and accuracy?
Also, do you think Siri was better at launch than she is now or vice versa?
Note: Siri is slated to get improvements with the release of iOS 6 this fall. APple previews Siri’s upcoming new abilities here.
Related articles:
With Siri and new alliances, Apple takes on Google search – June 21, 2012
Nine auto makers partner with Apple for ‘Eyes Free’ Siri integration – June 12, 2012
Apple takes aim at Google with new in-house Maps with amazing Flyover, even smarter Siri – June 12, 2012
Apple files motion to dismiss Siri lawsuits – May 16, 2012
Apple hit with second class-action lawsuit over Siri intelligent personal assistant – March 30, 2012
Apple files Siri trademarks, faces lawsuit alleging ‘misleading’ Siri ads – March 21, 2012
Siri is hot and sexy
I give her a B+ with one sentences questions or task. Prolly a C- when it’s more than a sentence. Maybe even a D.
I’d give it a solid B on the topics it knows about like timers, weather, reminders, and appointments. The random voice search for trivia is also good or better than Google voice search. All praise to Wolfram Alpha integration.
I’m looking forward to the improvements in iOS 6 that were demonstrated at WWDC.
Interesting report… but left me a little confused. I am not sure if the number of times that Siri “accurately resolved the request” included the percentage of attempts that Siri did not comprehend the question, or was the percentage that Siri understood the request… but could not answer the question.
I find it really pointless to review a public beta as a first release. Beta = half complete buggy program not final release. And the logic part of Siri is all in the data center. Other data centers havent been built yet (just plans for new ones) so billions of people are using one center
True. And as a beta, Siri is a pretty polished piece of technology. For Google beta usually meant “whatever the kids cooked up in their spare time”.
Doesn’t matter if Apple calls it beta, Apple plugged Siri in national tv ads with Samuel L Jackson, so it is fair game for reviews, they can’t have it both ways.
I give Siri a B for accuracy btw. It’s gotten much better in the last couple months though
Siri has always worked very well for me. As long as I’ve got a decent connection…she’s been rock solid. Very few times has she not taken the correct action.
I’ve met plenty of human beings who are worse at answering questions.
1
Siri understands me far better than the typical outsourced tech support person speaking to me from another part of the globe, even if English is his/her native language. Not that I fault those people. They try. Language is hard. Point is that Siri does admirably.
Just recently (hint, hint), I’ve found Siri to have taken a quantum leap in her ability to understand, resolve, and answer my questions. Just sayin’, wait a couple more months, folks!
agreed.
Since I haven’t tried Siri since the first few days of iPhone 4S ownership, and many failures, I can’t tell you.
You are fooling no one.
Well he claims ownership of the iPhone 4S, so Apple has already gotten his money!
And as we all know, “A fool and his money are soon parted!” Right Sucker?
For both comprehension and accuracy, I’d give Gene Munster’s Apple analysis a D.
Hey Gene: how’s that Apple television you’ve been predicting for the last 2 years coming along?
a bunch!
I would give many people a below average grade on knowing how to use Siri’s current abilities to vastly improve their day. Your success is raised if you read some tips about what kinds of things to ask and how to structure your requests. Many people don’t take advantage of geo fencing for example. Or reminders, calender, alarms, etc.There are so many ways to use her to improve your productivity and efficiency that once you integrate her into your life within her current limited (Beta) abilities, you really would miss having her. and over time all this will improve and expand and become a crucial and major part of the future of intuitive user Interface. also currently back round noise will affect the level of her comprehension.
What is wrong with you, both cheerers and haters. Google understands 100% of the questions because it was typed in. What kind of test is this? Garbage in, garbage out.
Lmao.
Siri works great for me.
As with all voice recognition, you can’t drop letters from the word and expect it to be correct.
I found that you do have to talk at a constant pace and think about the words you use, and how you pronounce them.
I don’t use the dictation part of Siri that much, but the above was brought to you by Siri.
So shove that up your ass Jean.
(ok, so Siri got his name wrong.. Can’t get Siri to put gene.)
I use my iPhone as my alarm and often change the time. It’s great saying wake me at xxx. Works great. When I want to check weather it’s great to get a temp. These are small but helpful items for me. Texting goes up a level and has issues simple web requests have been super like are there any cxx around here so it depends I think apple may have been better to roll out limited functions slowly
I have to agree with Munster and give Siri a solid D. Siri struggles to find names in my Contacts, properly hear simple questions/answers to text messages, etc. I have completely given up trying to use Siri over Bluetooth in my car, which should be the most useful application of Siri.
I have noticed that Siri does not have all of the network outages she used to have. However, I still think it is ridiculous that Siri needs a network connection to look up someone in my Contacts, create a Calendar event, or do other items which are local to my iPhone and do not need network access.
I really hope Siri improves dramatically with iOS 6.
Queries: C- or D+. When it works, it works great. When it doesn’t it doesn’t. You know what I mean. Maybe she needs a kickstand. “We knew that if we didn’t get the kickstand perfect, the device would not work.” (Microsoft Surface keynote hilarity).
Dictation: A-. Love it for texting on iPhone 4S. Love it for entering complex numbers like UPS tracking numbers (don’t type it, just read it). Love it for “typing” on the new iPad!
Siri is not only below average, I’d giver her a solid F as in FAIL. She’s completely & utterly useless. She hasn’t understood me since day one and I keep her permanently switched off all the time as it’s just too much effort to try to communicate with her.
Useless piece of trash to goose the sales of the iPhone 4S.
If one agrees with your last statement, then Siri is fabulously successful in spite of its beta status. It did goose the sales. It also wasted tremendous amounts of time in workplaces as users demoed it to their friends asking silly questions.
Wait… So after all your carping about the 4S being a meaningless and unimportant release, you bought one? I thought you were holding out for a bigger screen… But then maybe I tuned out somewhere along the way… I use Siri about 30 times a day for various tasks (more, if you count individual snips of dictation), and her error rate is in the sub-10% range. Methinks you want her to fail.
Doesn’t Siri require frequent use to “get better?” Yes, I feel she was better at launch, and at certain things than others. She misses info I can get from Google. I’m hoping for the best- in the meantime, it’s a bonus for entertainment value.
For my use, I give Siri a good, solid ‘A’. She’s almost always available, and accuracy and such is at least 95%.
I have only had the 4S for just over a month but in about 3000 miles of highway travel, where I use it the most, I would give it an overall “B+” rating.
It finds restaurants ahead for me with reasonalble accuracy and with the need to repeat about 1 in 3 times.
It finds locations easily for me.
It finds phone numbers of companies and dials without a problem.
All of this is while I am driving and cannot look at the screen.
One prerequisite is to “understand” what it reponds to effectively and not leave out “understood” words.
My wife cannot get hers to work, but the difference is that she does not realize the difference in “understood/taken for granted” words versus speaking them. For INSTANCE: Once we were wanting a fast food restaurant (Popeye’s a cajun fast food restaurant). My wife asked Siri to search for “Popeyes” in X city. Me: “Find Popeyes restaurant in X city”
The difference is that “Stores, businesses, restaurants” are implied in human language and context to us, but when we go to Siri, It has to place the “words” into a HUGE context from which it cannot deliver precisely. Just adding “store” “business” “Restaurant” to a name clarifies it searches tremendously.
Other searches, a single word of context will greatly enhance its finds. Again, my wife does not think she should have to do this. AND before others specifically agree, I would ask someone from Maine or Washington State if they know what a “Popeyes” is. Context left out is a killer puzzle. All that it takes is a single one word “context” most of the time.
CLARITY:
Most people don’t like to think of themselves as poor speakers. I have a deep southern drawl but I also realize that I need to speak clearly in a steady pattern. I have had very little problem with Siri understanding and responding. My wife says that I change my speach patterns for Siri, and yes, I do. For my northern relatives, and CA relatives, I speak more precise and consistent. I don’t expect Siri to understand slow southern drawls with perfection. My wife disagrees. I get accurate results the first time at about a 90% rate, she gets results at about a 10% rate and her accent is much less than mine.
1
The clarity part is most people’s problem.
They think they are the most eloquent speaker, yet sound like krikpe on big bang theory.
How right you are. Just actually say those words that you wouldn’t bother with when talking to a person who already knows what youre thinking, and Siri’s effectiveness skyrockets. It doesn’t always have to be a specific word. Popeye’s chicken works just as well as Popeye’s restaurant. Just give her a little more to work with, that’s all she needs. Honestly, I’m stunned at how often she gets my drift when I’m already questioning whether I gave her enough info.
And now I must go find a Popeye’s. You’ve got me craving some popcorn shrimp.
Siri is rubbish to anyone outside of the US. ie. most users. Too often queries are met with: I can only look for businesses in the United States and when you’re using US English”
I use it constantly, very few errors and when it does happen I can easily attribute it to me slurring my words.
Also as was said above, it is getting even better (hint hint).
I do not have an iPhone 4S myself, but I was with somebody this past weekend who had one. We used it to ask it several trivia questions and it answered them perfectly. I was very impressed and jealous because I only have an iPhone 4.(This was dictated using Siri on the iPad.)
I have my iPhone 4s for three weeks.
Siri and bluetooth do not work together.
If I turn on bluetooth in the car, Siri and sounds do not work.
When I am home I miss calls and text unless I turn off Bluetooth when I get home.
When will Siri grow up?
Don’t tell me that: I had thought for sure it was working perfectly in my car (Some refurbished Boss DIN thing I put in a little over a month ago). I drove across the country talking to my Boss unit who picked up my speech and forwarded it via Bluetooth to my iPhone 4S where Siri took care of everything. Road noise didn’t bother it, and distance from my driving position to the mic on the Boss (pretty low in my car actually). So in my case, not only is ignorance bliss, but ignorance made it just work.
Very weird comparison. It seems to me it is kind of like saying that airplanes get a “Below Average” rating on ability to recognize landmarks from the window, as compared to driving in a car. Ok, yes. That’s true but um, YOUR FLYING!! Why compare voice entry to text entry? Different entry method, different results. How about comparing voice entry to text entry on number of horrible accidents avoided by not trying to do text entry while driving? This is not an “Apples” to apples comparison.