Watch all the Transform 2020 sessions on-demand here.
It’s a well-established fact that reviews impact sales. About 71 percent of customers say they’re “more comfortable” purchasing a product after reading other folks’ impressions of it, while 88 percent say reviews influence their buying decisions. There’s a real incentive, then, to cut down on spammy and manipulative reviews written by malicious third parties. Fortunately, scientists at the Hartman Group and the University of Washington have made progress in this challenging arena.
In a paper published on the preprint server Arxiv.org (“GANs for Semi-Supervised Opinion Spam Detection“), the researchers describe “spamGAN,” a generative adversarial network — i.e., a two-part neural network consisting of generators that produce samples and discriminators that attempt to distinguish between the generated samples and real-world samples — that learns from a limited corpus of annotated and unannotated data.
“Opinion spam is a widespread problem in ecommerce, social media, travel sites, [and] movie review sites,” the paper’s authors write. “Opinion spammers try to exploit such financial gains by providing spam reviews which influence readers and thereby affect sales. We consider the problem of identifying spam reviews as a classification problem, i.e., given a review, it needs to be classified either as spam or non-spam.”
As the team explains, spam reviews are a bit of a challenge to identify because of the relative dearth of prelabeled data sets. So the approach leans on a technique called semi-supervised learning, in which unlabeled data is used in conjunction with small amounts of labeled data to produce an improvement in learning accuracy.
June 5th: The AI Audit in NYC
Join us next week in NYC to engage with top executive leaders, delving into strategies for auditing AI models to ensure fairness, optimal performance, and ethical compliance across diverse organizations. Secure your attendance for this exclusive invite-only event.
Specifically, the generator component of spamGAN outputs new sentences — given a class label as input — that the discriminator attempts to classify as “real” (sampled from the training data set) or fake (unrealistic). Its performance is used as feedback to improve the generator, with better classification accuracy resulting in more rewards.
Prior to adversarial training, the researchers pretrained the descriminator with batches of labeled real sentences, unlabeled real sentences, and fake sentences. Next, they fed spamGAN 80 percent of a corpus of 1,597 reviews tokenized at the word level, comprising roughly 800 “truthful” reviews of Chicago hotels from TripAdvisor and 800 “deceptive” reviews crowdsourced from Amazon Mechanical Turk. (The remaining 20 percent was used to validate the AI model’s performance.)
In experiments repeated 10 times each, the researchers say that spamGAN outperformed existing spam detection techniques with limited labeled data, achieving accuracy (measured by F1 score) of between 71 and 86 percent when trained on as little as 10 percent of labeled data. Apart from detecting spam reviews, spamGAN managed to generate reviews, albeit with a degree of perplexity, like: “Loved this hotel but i decided to the hotel in a establishment didnt [sic] look bad …the [P]almer house was anyplace that others said in the reviews …”
The researchers leave experiments with other review data sets and with a more sophisticated classifier to future work.
“[M]ost existing research on opinion spam (other than deep learning methods) manually identify heuristics [and] features for classifying spamming behavior; however, in our GAN-based approach, the features are learned by the neural network,” they wrote. “[We believe that] spamGAN can also generate spam [and] non-spam reviews … which can be used for synthetic data generation in cases with limited ground truth.”
