“A picture is worth a thousand words” could very well describe the current search paradigm. We’ve spent years typing into a text field, but there are times when it’s hard to describe what we’re really looking for.
Traditional search engines are built around intent, but they usually fall short when it comes to discovery. This is an area where Pinterest believes it can shine. In the past couple of years, the company has sought to redefine the future of retail, one pin at a time.
Everything in the world is a pin
In 2015, Pinterest revealed it had been working on a way to find and display visually similar images, which came out of its acquisition of VisualGraph a year prior. Initially, visual search only powered the company’s Related Pins feature, but now is more widely used, leveraging deep learning to train itself and extract information from images. “We try to understand what’s in an image,” Kevin Jing, the former head of the company’s visual discovery team, told VentureBeat in an earlier interview.
It seems Pinterest’s work is starting to pay dividends. “Right now, it’s being used by tens of millions of people every month,” Jeff Harris, the company’s visual search product lead, told us. “There are over 300 million visual searches monthly, so I feel like in terms of computer vision at scale, it’s one example that’s kind of a breakaway success.”
June 5th: The AI Audit in NYC
Join us next week in NYC to engage with top executive leaders, delving into strategies for auditing AI models to ensure fairness, optimal performance, and ethical compliance across diverse organizations. Secure your attendance for this exclusive invite-only event.
Above: Pinterest visual search in Firefox.
Confident in its ability to identify a purse, bar stool, dress, or anything else within an image, the company introduced its Lens technology, which has the potential to be a major influence in how we shop today. With a tool that retailers could use within their mobile apps, Pinterest not only provides creative options for outfits, accessories, styles, and choices consumers never really had in real time before, but it also helps the company improve its algorithm.
“I think the goal [of visual search] is that the whole world should be pinnable,” Harris said. “You see a lot of cool things on Pinterest, but you also see a lot of cool things in the real world, and if you see lighting in a restaurant that you’re enamored with, what do you do with that? How do you act on that? We think that Lens is a way to make the whole world interactable and connect that world with all the stuff that exists online.”
Naturally, Pinterest launched its Lens tool to the public within its own apps, but the real success came when the company struck an exclusive retail deal with Target and also became a major partner with Samsung when the phone maker launched its Bixby visual search.
Above: Pinterest visual search within Target’s mobile app.
Harris believes it’s too early to tell if users need to be educated about the potential of visual search, but he added that right now Pinterest is focused on promoting really strong use cases. Target and Samsung, for example, already had “a lot of eyeballs” and “people are in a mindset where visual search would help.”
“I really think that the way this is going to work for us is if we can meet people in the moments where they have an implicit visual question in their head and they don’t necessarily need to think ‘Hey, I’m going to go over to Lens and do it.’ No, they were already doing a fashion search. They were trying to figure out an outfit from their wardrobe. If the camera pops up with a little toast that says ‘Hey, take a picture of the piece of clothing that you want’, I think that’s a really good moment to educate people,” Harris said.
Above: Pinterest’s Shop the Look feature.
He referenced the company’s recent launch of its Lens Your Look feature, which is intended to capitalize on the 16 billion pins that are fashion-related (out of 100 billion total pins). “Fashion is one of the top things that people come to us for, and what we added was just a camera functionality to fashion searches, instead of just limiting yourself to text,” Harris explained.
Pinterest’s strategy is to be deliberate in its rollout of Lens to all 200 million users. “I’m not at a place where I think we need to push Lens on all Pinners. I think we should figure out the core things that we’re really good at and make sure that the people who are already using Pinterest for that see it that way,” Harris said.
The future of retail
Pinterest isn’t alone in its pursuit of visual search as a means to reimagine retail. Google, for one, is moving along similar lines. In May, the Alphabet-owned company launched its own version of Lens, which would be available through Google Photos and Google Assistant. Tapping into these applications would give Google a tremendous advantage because its reach is so much larger than Pinterest’s, even with the latter’s partnerships. But the Lens feature in Google Assistant is unfortunately limited to Pixel phones, where it only starting popping up earlier this week.
Harris believed that Google’s foray into visual search isn’t necessarily a threat to Pinterest, suggesting that research being done in this area benefits everybody.
And he believes that his company holds an edge: “Right now, most of Pinterest is built on this idea that ‘I like pins A and B, you like pins A, B, and C, so I’m probably going to like pin C.’ That’s sort of how all the logic is built, but if a human was trying to give you style advice or interior decorating advice, they wouldn’t use any of that collaborative filtering-type stuff. They would say ‘Oh, you like Ward tables that are made of walnut’, or ‘Your fashion sense is you really like muted colors with a particular kind of tie.'”
Pinterest already has massive amounts of data, but understanding it requires computer vision. As that technology is applied over time, Harris is confident that the experience will continue to improve. “It’s going to make the relevancy across our entire product so much better and [it will] really start to understand your taste in an aesthetic way. That’s going to make us the best stylist, the best interior decorator, etc. because not only do we have the vision to see what you’re looking at, but [we] also have 16 billion style outfits, including the latest trends, so there’s a lot of information.”
Above: Pinterest Lens.
While Pinterest currently uses its Interest or Taste Graph to surface pins you might be interested in, Harris said the next step involves making matches based on what you like in the real world.
“All of that is going to take computer vision, and I think that it’s going to be just as powerful as the Interest Graph, if not more so, because these are actual aesthetic decisions that people have chosen to design their room, their outfit around, and that data is completely invisible to computers today,” he said. “Computer vision is going to unlock that, and we have this unseen catalog of images of people’s taste decisions that they’ve already made. We just need to be able to process it.”
Where does Pinterest go next?
We asked Harris how he believes visual search is evolving. He said, “If you think about how search works across the web and all these platforms, it’s almost entirely limited to just words. I think that’s really unnatural — when I talk to you, I use words, but I also show you things, and that’s how humans converse with each other.
Above: An example of Pinterest’s Lens feature.
“And it’s just one medium that you’re forced to communicate through. We think that as search evolves on Pinterest, it’s going to be more and more like human conversation, where there are some queries that are going to make the most sense as words — you can just cleanly describe them in words. But some queries, like ‘I want a couch that fits into this living room’ — good luck describing what your living room is in words, it’s really hard. Taking a picture of that room can make that query much easier.”
Harris said that to better serve users, Pinterest has to understand searches in which images and words are intermixed.
In terms of progress around visual search, Harris believes Pinterest is just getting started. He said that the company is looking at improving product identification so that if you see something in the real world and want to find that specific item, you can simply use a Pinterest Lens-powered app.
Pinterest is also interested in becoming a more proactive virtual personal shopping assistant. If you buy a purse, for example, Pinterest could tap into its search smarts to surface related outfits that would go with it. Ultimately, Harris believes Pinterest will be able to answer a range of visual questions people have, such as what makeup to buy or whether a particular piece of furniture will work in their home.
“Your brain has evolved for so long to be really good at knowing what everything around you is. That is not where computer vision is going to add value,” Harris explained. He predicted it will come into its own when it can make suggestions based on information you didn’t know or didn’t think to ask.
