Watch all the Transform 2020 sessions on-demand here.
What do chipmakers Qualcomm, Huawei, and MediaTek have in common? All three market hardware architectures to accelerate computer vision, natural language processing, and additional machine learning tasks in smartphones, tablets, and other mobile devices. The trouble is, these are a challenge to benchmark empirically, which can make it difficult for developers to optimize algorithms for particular platforms.
Researchers at ETH Zurich are attempting to tackle the problem with a new app for Android devices, called AI Benchmark, which they hope will become a universally accepted test suite. It launched today on the Google Play Store and works on any handset running Android 4.1 or higher.
“Since there was no information about this (currently, all AI algorithms are running remotely on the servers, not on your device, except for some built-in apps integrated in phone’s firmware), we decided to develop our own tool that will clearly show the performance and capabilities of each device,” AI researcher and developer Andrey Ignatov told TechCrunch.
Above: AI Benchmark running through computer vision tasks.
AI Benchmark evaluates smartphones’ performance across a suite of open source algorithms that perform image classification, facial recognition, image super-resolution, and photo enhancement, segmentation, and deblurring. It even tests performance on neural networks used in driverless cars, which the researchers predict might eventually run on chips comparable to those found in smartphones.
June 5th: The AI Audit in NYC
Join us next week in NYC to engage with top executive leaders, delving into strategies for auditing AI models to ensure fairness, optimal performance, and ethical compliance across diverse organizations. Secure your attendance for this exclusive invite-only event.
As the app puts the phones through their paces, it produces visualizations of the algorithms’ output and spits out a score that factors in the speed of the system-on-chip and available RAM. (Generally speaking, the larger the neural network, the more RAM is needed to process it.)
Here’s a partial list of the top performers, along with their associated scores. A full list is available on the AI Benchmark website:
- Huawei P20 Pro – 6397
- OnePlus 6 – 1875
- Sony Xperia XZ2 – 1664
- Samsung Galaxy S9+ – 1494
- Razer Phone – 1470
- Samsung Galaxy S9 – 1446
- OnePlus 5T – 1440
- OnePlus 5 – 1416
- Samsung Galaxy Note 8 – 1408
- Xiaomi Mi Mix 2 – 1405
So how do the top smartphones compare? Ignatov summarized the team’s initial findings as “both interesting and absurd.”
Qualcomm chips, he said, accelerate quantized — or compressed — neural networks that aren’t compatible with all versions of Android. Typical networks that don’t use the company’s proprietary Snapdragon software development can’t take advantage of its Hexagon DSP AI acceleration chip, though Ignatov notes that such networks could benefit if the “appropriate drivers” were implemented.
“[If] you are developing an app that is using AI, you won’t get any acceleration on Snapdragon’s SoCs unless you are developing it for their processors only,” Ignatov said.
Huawei’s Kirin Neural Processing Unit (NPU), on the other hand, offered great performance — almost 10 times acceleration for neural networks — but isn’t compatible with quantized neural networks yet. Huawei says support will come later this year. (Ignatov noted that the Huawei P20 and P20 Pro are the only phones on the market running Android 8.1 Oreo that provide AI acceleration to applications.)
As for Samsung’s Exynos lineup and MediaTek’s NeuroPilot AI platform, the results were a bit of a mixed bag.
Samsung’s processors only support AI acceleration starting from Android 8.1, and performance varied “significantly” — up to 50 percent in some cases — as a result of power optimization and throttling.
Meanwhile, MediaTek’s chipsets — among the few that support both quantized and normal neural networks — were outperformed by Samsung and Huawei’s hardware.
“In summary: Snapdragon — can theoretically provide good results, but are lacking the drivers; Huawei — quite outstanding results now and most probably in the nearest future; Samsung — no acceleration support now (most likely this will change soon since they are now developing their own AI Chip), but powerful CPUs; Mediatek — good results for mid-range devices, but definitely no breakthrough,” Ignatov told TechCrunch.
