Watch all the Transform 2020 sessions on-demand here.
Researchers from Google AI (formerly Google Research) and Toyota Technological Institute of Chicago have created ALBERT, an AI model that achieves state-of-the-art results that exceed human performance. ALBERT now claims first place on major NLP performance leaderboards for benchmarks like GLUE and SQuAD 2.0, and high RACE performance score.
On the Stanford Question Answering Dataset benchmark (SQUAD), ALBERT achieves a score of 92.2, on General Language Understanding Evaluation (GLUE) benchmark, ALBERT achieves a score of 89.4, and on ReAding Comprehension from English Examinations (RACE) benchmark, ALBERT gets a score of 89.4%.
ALBERT is a version of Transformer-based BERT that “uses parameter reduction techniques to lower memory consumption and increase the training speed of BERT,” according to a paper published on OpenReview.net Wednesday and on arXiv on Thursday. The paper was published alongside other papers being considered for publication as part of the International Conference of Learning Representations, which will take place in April 2020 in Addis Ababa, Ethiopia. ICLR will be the first international AI community conference held in Africa.
“Our proposed methods lead to models that scale much better compared to the original BERT. We also use a self-supervised loss that focuses on modeling inter-sentence coherence, and show it consistently helps downstream tasks with multi-sentence inputs,” the paper reads.
June 5th: The AI Audit in NYC
Join us next week in NYC to engage with top executive leaders, delving into strategies for auditing AI models to ensure fairness, optimal performance, and ethical compliance across diverse organizations. Secure your attendance for this exclusive invite-only event.
ALBERT is the latest derivative of BERT to claim a top spot in major benchmark tests. In late July, Facebook AI Research introduced RoBERTa, a model that achieved state-of-the-art results, and in May, Microsoft AI researchers introduced Multi-Task Deep Neural Network (MT-DNN), a model that achieved top marks in 7 of 9 GLUE benchmarks.
Each of the models achieves performance that outpaces average human performance.
In other Transformer-related news, Hugging Face, a startup whose PyTorch library for easy use of major Transformer models like BERT, Open AI’s GPT-2 and Google’s XLNet today made that library available for TensorFlow. PyTorch-Transformers has seen more than 500,000 Pip installs since the start of year, Hugging Face CEO Clément Delangue told VentureBeat.
