Watch all the Transform 2020 sessions on-demand here.
Though it’s easy to knock Apple’s AI assistant Siri for mediocre overall performance, some of its features are indeed improving — though others continue to struggle. An Apple Machine Learning Journal post today highlights recent under-the-hood enhancements to Siri’s recognition of local points of interest (POI), which the company notes up front “has remained a performance bottleneck” with high levels of inaccuracy — in part due to “millions of smaller, local POIs.”
The good news is that Apple has figured out that it can increase Siri’s speech recognition accuracy for local POIs by leveraging location data. Specifically, after determining an iPhone’s current longitude and latitude, Siri can then cross-reference a POI database corresponding to one of 169 U.S. geographic regions, matching a user’s request against locally available options. On the server side, Apple trains a machine with 5,000 hours of speech data and uses a manageably small, easy-to-update language model that can be frequently updated with new POIs as businesses open and close.
Across a number of city-wide tests, Apple was able to reduce POI identification errors for “the most popular POIs” by as much as 48.4 percent. In the most extreme case, Los Angeles’ “popular POIs” error rate dropped from 24.4 percent down to 12.6 percent. In other cities, such as New York, the reduction was closer to 42 percent, with a 15.7 percent remaining error rate. Using location data didn’t damage Siri’s ability to process non-POI requests, either.
June 5th: The AI Audit in NYC
Join us next week in NYC to engage with top executive leaders, delving into strategies for auditing AI models to ensure fairness, optimal performance, and ethical compliance across diverse organizations. Secure your attendance for this exclusive invite-only event.
Of course, there’s some bad news, too. Even with a reduction of that magnitude, Siri will still misidentify one “popular POI” for every seven or eight requests. That‘s better than before — one in four — but still annoying when you’re trying to get directions in your car. And in real-world user tests with larger test sets, the new system only achieved an 18.7 percent overall error reduction, while adding a little additional latency.
Improving Siri’s performance is particularly challenging for Apple due to its user privacy and on-device processing philosophies, which put it at a disadvantage compared with cloud-focused and less privacy-concerned rivals such as Google and Amazon. While the assistant still has a long way to go, it’s reassuring to see that it’s at least making some gains.
