Gene Munster, everyone’s favorite Apple television cheerleader, decided to take a day off from being wrong about the company’s future products and spent the day being wrong about their current offerings. As reported by Bloomberg (obviously Munster wouldn’t make Piper Jeffray money if he gave his priceless analysis away), he did some extensive testing of Siri and found it to be seriously lacking, an assessment I’d apply to Munster’s analysis in general. In the end, he recommended that Google does a better job understanding queries and returning relevant results and gave Siri a score of B for “comprehension” and a D for “accuracy”. Not to be a nit – actually precisely to be a nit – Munster should check out a dictionary before he publishes something. Comprehension, as in, “to understand the nature or meaning of; grasp with the mind; perceive” is what Munster wants to slag with his bullshit “grading”. Accuracy, or “the condition or quality of being true, correct, or exact; freedom from error or defect; precision or exactness; correctness” is something Munster admits Siri does quite well – over 80% according to his tests. I will say using words that “don’t mean what you think they mean” is pretty much in line with what I’ve come to expect from Munster’s scrawling.
Munster’s methodology? Oh that. He tested his iPhone 4S devices in both “a quiet room” and on the streets somewhere in Minneapolis, his team speaking and dictating to their handsets 800 times in each scenario. I guess it must still be winter in Minneapolis. I will say his “n” was more impressive than Munster’s last statistical opus where he asked 100 developers – at WWDC no less – about their preferred platform and then wrote a story about it. Statisticians around the world use stories about Munster’s methodologies to frighten their children into doing their math homework.
When he compared Siri’s results with Google’s, he found that Google “comprehended” his queries 100% of the time and returned higher quality responses, earning it the grade of B+. Google’s results were typed in, which is hilarious since the iPhone Google app has a voice input option. So why wasn’t there parity on input? We’re dealing with Gene Munster here. That’s about as good of an answer as I can give you. Siri understood 83% or 89% of the questions it was asked, depending on whether the scenario was “noisy” or “loud”. Google’s “accuracy” was 86% – or a B+ for those of you who need to have a dumb comparison dumbed down further. Siri’s “accuracy” was 62/68%, earning her a D. Munster didn’t grade how well his testers were able to make appointments, set timers or reminders, send text messages and call people using both services -the things people actually use Siri for – because that would have made Google look dumb.
Gene Munster’s point – if he has one beyond embarrassing himself with his crap surveys – is that Siri isn’t Google search – yet. Shocking, I know. I haven’t seen the Siri commercial where Samuel L. Jackson asks Siri who Peyton Manning plays for or John Malkovitch asks where Elvis is buried, so I’m pretty sure that’s not how Apple is advertising it either. But instead of testing how Apple bills the beta of its assistant, Munster would rather rig a test and compare one of its features to Google’s strongest. That’s as stupid as claiming that there’s an Apple television in the pipeline any day now, which happens to be Gene’s other specialty.
RSS
Twitter
