“Google’s Knowledge Graph has been the center of much attention lately. We have been hearing a lot about another concept called the Knowledge Vault as well. But just how extensive is Google’s capability? And what about Siri and Bing/Cortana? How do they stack up?” Eric Enge writes for Stone Temple Consulting. “These are the things we set out to measure in this study. To do that, we took 3086 different queries and compared them across all three platforms. These were not random queries. In fact, they were picked because we felt they were likely to trigger a knowledge panel.”
“In addition, this was a straight up knowledge box comparison, not a personal assistant comparison. For purposes of this study, a ‘knowledge box’ or ‘knowledge panel’ is defined as content in the search results that attempts to directly answer a question asked in a search query,” Enge writes. “All queries in this test were done using voice commands, even when using Google and Bing. The reason we did this is that there are many commands in Google and Bing that behave differently when the search query is typed in, and we wanted to do a straight apples to apples comparison.”
“Google Now returns twice as many results as Siri and nearly three times as many results as Cortana,” Enge writes. “This is clear evidence that Google is much further down the path with this type of work than either Apple or Cortana. As noted above, Bing, using text-based search queries, returns knowledge boxes for more types of results than Cortana does at this time.”
Read more in the full article here.
MacDailyNews Take: Apple clearly has much work to do, but note that iPhone, iPad, and iPod touch users can really supercharge their knowledge box results with the free WolframAlpha Viewer which allows users to explore Wolfram|Alpha results from Siri.
A more comprehensive test of knowledge boxes would have graded Siri with the WolframAlpha Viewer installed which would have significantly improved Siri’s results.
[Thanks to MacDailyNews Reader “Edward W.” for the heads up.]
Google’s list “it’s been way too long”. More hits doesn’t equal more tasty.
The test is to show how many of the 3086 questions resulted in “Knowledge boxes” being returned by each voice search, not how many results it returned. So in this case more ‘correct’ is better.
I’ve found Google’s search results to be less than useful. . . with too many paid advertising listings heading the top of the returns. Truly useful responses are often buried multiple responses deep in junk.
Reading and comprehension are not required to post here! Type anything that crosses your mind when you read an article title and don’t let the facts get in your way.
Brilliant.
I am suspicious of how this was compiled. Part of what makes a good search tool is how many other people are searching for similar data. There are FAR more Apple users than Android even with the glut of Android phones. It has been shown time after time that Apple users spend more and surf more on the web. Raw data searches may actually skew the results to Android, but in real world searches, Siri might actually perform better. There is no way to tell from what I see on this test. As far as I am concerned, the jury is still out and I am betting on Apple to end up with the better product. More hits? On this test maybe, but not necessarily in the real world….
Considering that the Google Search DB which Google Now is dependent on has a much higher use rate and longer history compared to Safari/Siri I’m not surprised Google would have compiled a more comprehensive DB..
Google Now look pretty good except for the Google part.
I’m in the process of de-Googleizing.
Who cares how many results it returns. I am less and less happy with Google regarding a search result. It’s not always accurate and if you don’t have the result on the first page it is not getting better in the next pages. Unless you are looking for a listing like all realtors in my city or something very broad but if you are looking for something more specialized it’s getting worse. Apple and Bing are not even there yet.
I was wondering the same thing.
During the Maps mess up, people noticed that Apple Maps was more likely to not give an answer when it didn’t know, while Google Maps was more likely to give an answer even if it was nonsense.
Exactly. Numbers don’t matter. The quality of the search result does.
A lot of people are misunderstanding the excerpt from the article. It’s not the number of results returned for each search that they’re counting. It’s how many queries are being answered. According to their tests, Google provided an answer to more of the queries.
And it’s not pages of results. It’s a knowledge panel (or speech feedback).
As MDN said, Apple has its work cut out for it.
This is actually really complex stuff compared to web indexing and Google has been doing it for a long time. It involves numerous sources of data and a curation of the potential queries and relevant answers.
Apple will get there, but it will take time. It’s a lot like Maps. It requires a ton of manual hours of properly entering and editing the data.
Yep, de-google complete. At least I don’t feel like I have to take a shower after reading my gmail.
I don’t have time to read the whole article right now, are they basing the results on the *number* of results?
Quantity does not equal quality..
So you go to the article..
“Not only do we get our answer, but Google offers us info on three other tall buildings. Google noticed that people who search on the height of the Eiffel Tower often want to know the height of other tall buildings. If you click on the “Burj Khalifa” link, it becomes even more interesting. Here is what you get:”
So Siri/Cortana/Google ALL give the same answer for the question you asked… but Google gives you the height of buildings you DIDN’T ask for as well?
so Google wins that question?
If you asked how tall the Eiffel tower was COMPARED to other tall buildings… then yes, Google would win that question.
But thats NOT what was asked.. Maybe i’m standing in front of the tower and am curious, I don’t give a crap what other buildings heights are.
For that question, I don’t see google as “winning” that one.. surprised the result didn’t place Google HQ in the list. Or try and sell you on an Eiffel tower trip..
I don’t think the ‘win’ for each question was exclusive.. All 3 got one point for resulting in a Knowledge Box.. The author simply noted there was extra relevant info based on the number of people that proceeded to ask about other buildings. I have tried the query and the result was a knowledge box with the height of the Eiffel Tower, voice reading the answer and the lower part of the box showing heights of relevant other buildings which were not read.
Siri is pretty stupid and consistently so. Cue Siri Saturday with NCAA Football scores and you will get a hodgepodge of games in no recognizable order. Try different phrasing and you will get other hodgepodges, but no properly organized results.
I asked, “Siri, what time is the Seahawks game today?”
She gave me the answer.
I then asked: “which channel is carrying the game?”
She gave the right answer.
That ability to infer important details based on previous dialogue is what we call a conversation. And while no doubt simply implemented and quite limited it is nonetheless impressive.
I am not overly impressed with Siri but the new iOS typing capabilities are pretty good and I like how quickly it types …..
Everyone expects Apple’s products to be perfect but if you remember Siri was beta for long time and is still a work in progress, maybe they should bring back the beta status with the results as tested ……
Nah, just teasing but Apple does have a ways to go on Siri!
They’ve never said Siri was out of beta.
I guess the assumption is that if Apple is implementing it and there is no mention otherwise, the feature is not in Beta. Google had Gmail in Beta for a long time and you knew it since it said so in the Gmail logo. Doubt Apple would do the same by placing a small beta mark on the screen somewhere whenever Siri is active.
Pretty impressive how far they are ahead of Microsoft really, when you consider these results are more attuned to a search engine knowledge base than a personal assistant.
A test based on the number of results returned is useless unless those results are valid and impartial. By the testing criteria used here, if a search engine returned the whole internet every time it processed a query it would be rated #1. The whole point of a query is to return a handful of RELEVANT results. Returning irrelevant results or results that paid a fee to be included counts as a negative, not a positive. When I search for something I want the 10 to 20 most relevant results, not 200. A perfect search engine would give you the one most relevant result per query.
I think you misunderstand the test.. 3086 questions were asked and the number of times the result was a knowledge box was counted. Not the number of results total for questions asked..
Yea, I frequently look at 1,800 pages with each search I perform.
Do you know of anyone who ever looks beyond page one with search results?
When you do, you find out quickly how useless pages 3 and 4 are. (yes, I use Google search with laptop Safari)
That’s great for Google, but I prefer not to have a little piece of my soul sucked away by Google with every search I perform . . . .
@MDN: That’s great about the WolframAlpha Viewer app, but seeing as this is the first I have ever heard about it, what is the likelihood that most iPhone users have heard about it either? And why do we need another app to give us better Siri results?
It’s not better Siri results, it’s being able to see Siri results in a better format.
Found another article that covers the same comparison but also claims that “Siri uses Wolfram Alpha as its data source, “.. So not sure if adding the App to the iPhone will change the results.
http://news.softpedia.com/news/Google-Now-Beats-Siri-and-Cortana-to-a-Pulp-in-Comparison-Study-461713.shtml
I went to website and found this:
For more information on Google+ Services, contact us at:
Stone Temple Consulting
I guess I might question the impartiality of this test.
Also, did they ask real life questions like where can I eat around here, what movies are nearby, etc, or “knowledge vault” questions most people don’t want to find out on their phone, like who won the 1923 World Series, what were the names of all the Marx brothers, etc. Their example of “how tall is the Eiffel tower” leads me to think they didn’t ask what most people want to know at the time. I want Siri to give me real world info, not play Jeopardy with me.
Good question.. Also they could have had related follow up questions in the bunch.. For example the following sequence: 1) Who is the president of the United States 2) Who is his wife 3) What are their children’s names. Google Now is developed enough to actually answer the sequence of questions correctly.
Asked Siri the following:
Distance to moon
atomic weight of copper
melting point of copper
half-life of lawrencium
distance to Los Angeles, (did’t tell Siri from where, wanted her to assume from Chicago)
distance to Highland Park, (Illinois)
my business phone number
when was Albert Einstein born
whose playing the Blackhawks tonight
who is the goalie for the Hawks
what are the names of my children
what’s my wife’s name
how many days till Christmas
All correct, no mistakes. Wrote all queries on the fly, almost as fast as I spoke. Almost all answers instantaneous.
“what are the names of my children”,
“what’s my wife’s name”
are good examples when you’ve already told Siri who they are but is going to be incorrect if you hand your phone to a married friend with kids to ask those same questions. Unless that is Siri can distinguish between voices. 😀
I would humbly suggest that if you don’t know the names of your wife and children that you have more serious problems than you realize. Consider Siri’s advice to contact a marriage counsellor.
I understand that those types of personal questions are unique to Siri currently.. Probably kept in the cloud attached to your account after you teach Siri.
Siri needs to catch up , google gives much better results and I just hate freaking google. I asked siri, which is supposed to have shazam built in to identify 3 songs, 2 of which where elvis songs. She got none. The actual shazam app got all three. She needs work.
That’s strange. I’ve tried Siri on song identification quite a few times since upgrading to iOS 8, and she’s never missed, even in a noisy car, or listening to music under a voice over.
You have to have the app installed, and configured for Siri to use it.
I Know Soundhound works, is Shazam listed as a Siri Extension?
That’s an interesting feature of Siri.. I wonder what happens when you install all such extension apps. Do you end up with multiple knowledge boxes when there are multiple extension app databases that have relevant info?
Siri cannot figure out “Yoshinoya”, try asking for directions to that.