# What We Learned from Using LLMs in Pinterest — Mukuntha Narayanan, Han Wang, Pinterest AI Engineer · 2025-07-16 Pinterest search engineers Han Wang and Mukuntha Narayanan present four key learnings from integrating LLMs into the platform's search relevance system. Fine-tuned LLM cross-encoders improve relevance prediction by 12% over multilingual BERT and 20% over SearchSage using an 8B model. VLM-generated image captions and user action features enrich pin text representations and further boost performance. Knowledge distillation into a bi-encoder student model, trained on 100x more data via semi-supervised learning, enables production serving with real-time query embeddings and 85% cache hit rate. Relevance-tuned embeddings serve as general-purpose semantic representations across multiple surfaces, and the system yields relevance gains in multiple languages and countries despite a predominantly US training set. ## Questions this episode answers ### What performance improvements did Pinterest achieve by using LLMs for search relevance? Han Wang shared that an 8-billion-parameter LLM improved relevance prediction by 12% over a multilingual BERT model and 20% over Pinterest's in-house SearchSage embedding. Larger, more advanced LLMs yielded even better results, demonstrating LLMs' strong capability in predicting relevance on a 5-point scale. [4:07](https://aie.addtry.com/e05f72e8-c5c3-4688-9bbf-74766de4508d?t=247000) ### How did Pinterest enrich content representation with visual and user-action signals for relevance scoring? Han explained they built text representations using VLM-generated synthetic image captions, plus user engagement features like board titles and high-engagement search queries. Ablation studies showed adding these features sequentially improved performance, with user-action annotations notably boosting the model's content understanding. [5:05](https://aie.addtry.com/e05f72e8-c5c3-4688-9bbf-74766de4508d?t=305000) ### How did Pinterest productionize large LLM relevance models at scale without high latency? Mukuntha described distilling the teacher LLM into a bi-encoder student using semi-supervised learning on daily search logs, scaling data 100x. Offline pin embeddings are cached; real-time query embeddings are inferred with an 85% cache hit rate. This lifted relevance NDCG and search fulfillment internationally, even outside the US-labeled data. [7:23](https://aie.addtry.com/e05f72e8-c5c3-4688-9bbf-74766de4508d?t=443000) ### What additional uses did Pinterest find for the embeddings from their relevance-tuned LLMs? Mukuntha noted that the distilled model's pin and query embeddings served as rich, general-purpose semantic representations. They improved content representation across other Pinterest surfaces like related pins and the home feed, demonstrating that relevance tuning yields high-quality embeddings useful beyond search. [12:10](https://aie.addtry.com/e05f72e8-c5c3-4688-9bbf-74766de4508d?t=730000) ## Key moments - **[0:00] Introduction** - [1:15] Pinterest search processes over 6 billion monthly queries across 45 languages in over 100 countries. - **[1:52] Search Architecture** - **[2:29] Relevance Model** - **[2:55] LLM Performance** - [4:15] Han Wang: 8B-parameter LLM model improves search relevance 20% over SearchSage embedding and 12% over multilingual BERT. - **[5:05] VLM Captions** - [6:02] Han Wang: User action features like board titles and engagement queries improve relevance beyond synthetic image captions. - **[7:14] Knowledge Distillation** - [7:56] Mukuntha Narayanan: Semi-supervised labeling with LLM teacher on 100x daily search logs enables scaling to global, multilingual relevance. - [9:22] Mukuntha Narayanan: Bi-encoder with offline pin embedding and query caching achieves low latency with 85% cache hit rate. - [11:11] Mukuntha Narayanan: Distilled student model lifts search fulfillment and NDCG in Germany and France despite limited US training data. - **[12:10] Bonus Embeddings** - [12:10] Mukuntha Narayanan: Relevance-tuned embeddings from the student model improve home feed and related pins content representations. - **[13:14] Recap** - **[13:56] Q&A** - [14:04] Q: How did Pinterest select the open-source LLM to fine-tune? Han Wang: Extensive experimentation across models determined the best performer. - [15:14] Q: How does the LLM influence online search ranking? Mukuntha Narayanan: The teacher distills into a student that predicts 5-scale relevance scores used in re-ranking. - [16:38] Q: Is the embedding model multimodal? Mukuntha Narayanan: The team is exploring multimodal, but VLM captions currently extract image features effectively. - [17:49] Q: Did Pinterest use separate models for each language? Han Wang: A single multilingual LLM is used for all languages, leveraging cross-lingual transfer. ## Speakers - **Han Wang** (guest) - **Mukuntha Narayanan** (guest) ## Topics Fine-Tuning, Embedding Models, Inference Optimization ## Mentioned Pinterest (company), GraphSage (product), OmniSage (product), SearchSage (product) ## Transcript ### Introduction **Han Wang** [0:15] Hi everyone, um, thanks for joining the talk today. We're super excited to be here and share some of the learnings we have from integrating the LLM into Pinterest search. My name is Han, and today I'll be presenting with Mukundha, and we are both machine learning engineers from the search relevance team at Pinterest. So, start with a brief introduction to Pinterest. Pinterest is a visual discovery platform where Pinterest can come to find inspiration to create a life they love. And there are three main discovery surfaces on Pinterest: the home feed, the related teams, and search. In today's talk, we'll be focusing on search and, um, where the user can type in their queries and find useful, inspiring content based on their information need. And we'll share how we leverage LLMs to improve the search relevance. Here are some key statistics for Pinterest search. Every month, we handle over 6 billion searches, with billions of pins to search from, covering topics from recipe, home decor, travel, fashion, and beyond. And at Pinterest, search is remarkably global and multilingual. We support over 45 languages and reach Pinterest in more than 100 countries. These numbers highlight the importance of search at Pinterest and why we're investing in search relevance to improve user experience. ### Search Architecture **Han Wang** [1:52] So, um, this is an overview of how Pinterest search works at the backend. It's similar to many recommendation systems in the industry. It has query understanding, retrieval, re-ranking, and the blending stage, and finally produces relevant and engaging search feeds. And, um, in today's talk, we'll be focusing on the semantic relevance modeling that happens at the re-ranking stage, and share about how we use LLMs to improve the search relevance on the search feed. ### Relevance Model **Han Wang** [2:30] Okay, so, um, here's our search relevance model, which is essentially a classification model. Given a search query and a pin, the model will predict how much the pin is relevant to this search query. And to measure this, we use a 5-point scale, ranging from the most relevant to most irrelevant. Alright, um, now we're going to share some key learnings we have from using the LLM to improve search Pinterest search relevance. And here are four main takeaways that we would like to, um, go into more detail. ### LLM Performance **Han Wang** [3:12] Lesson 1: LLMs are good at relevance prediction. Um, so before I present the results, let me first give a quick overview of the model architecture that we're using. We contain the query and the pin text together and pass them into an LLM to get an embedding. So this is called a cross-encoder structure, where we can better capture the interaction between the query and the pin. And then we feed the embedding from the LLM into an NLP layer to produce a five-dimensional vector, which corresponds to the five relevance levels. And during training, we fine-tune some open-source LLMs using Pinterest's internal data to better adapt the model to our Pinterest content. And here, um, I'd like to share some results, um, to demonstrate the usefulness of LLMs. And as a baseline, we use SearchSage, which is a Pinterest in-house content and query embedding. And, um, so if you look at the table, you can see that the LLM has substantially, um, improved the performance of the relevance prediction. And as we use more advanced LLMs and increase the model size, the performance keeps improving. And for example, the 8 billion LLMs model gives, um, 12% of improvement over the multilingual birth-based model and 20% of improvement over the SearchSage embedding model. So, um, the lesson here is that LLMs, they are quite good at relevance prediction. ### VLM Captions **Han Wang** [5:05] Lesson 2: The vision language model-generated captions and the user actions can be quite useful for content annotations. So, to use LLMs for search, uh, for relevance prediction, we need to build a text representation of each pin. And here I listed several features that we used in our model. Besides the, um, the title description of the pin, we also include, um, the VLM-generated synthetic image caption to directly extract information from the image itself. And besides that, we add some, um, user engagement-based features, like the board titles, um, for the user-curated board that the pin has been saved to, or, um, the queries that led to the highest engagement with this pin on the search surface. So these two user action-based features, um, serve as additional annotations for the content. And, um, here the five sources of features together help to build a more, um, robust and comprehensive text representation for each pin. Um, to understand the, um, importance of each vertex feature, we also did some ablation studies. We used the, um, VLM-generated image caption as a baseline. And, um, as you can see, itself already, um, provides a very solid baseline. And as we sequentially add more vertex features, we keep seeing performance improvement. And this indicates that enriching the vertex features is quite useful for relevance prediction. And notably, um, the last two rows of the table show the performance gain we have by adding these user action-based features. So these features turned out to be quite useful content annotations that help the model better understand the content. ### Knowledge Distillation **Han Wang** [7:14] Alright, um, next I will hand over to Mukundha to talk about how we use knowledge distillation to productionize this model. **Mukuntha Narayanan** [7:23] Great. Yeah, uh, so now we have a good relevance model, which is good at predicting search relevance. Uh, but how do we actually scale this up without bankrupting Pinterest? Uh, usually the answer is knowledge distillation into smaller models. Um, and this is the production-served relevance-driven model that we distilled from the teacher model using semi-supervised learning. Uh, the student model is trained to predict five-scale relevance scores too. Uh, it trains using the five-scale soft scores produced by the teacher model. Um, and we produce data for this using a semi-supervised learning setup that, uh, I'll show in the next slide. So the LLM teacher model is trained on a small set of human-labeled data that we get from human annotators who are trained in very specific segments. Uh, we fine-tune, and this is a multilingual language model which uses pretty generic features which scale across a lot of different domains, etc. Um, and the way we get training data for the student, uh, is through, uh, sampling from daily search logs, which is, um, all the searches, uh, people make on Pinterest. Uh, and since we sample daily, uh, this includes any trending queries, all the latest freshest pins on Pinterest. And this is also remarkably go-global, like we mentioned, and only a small subset of this comes from the US, where most of our human-labeled data comes from. Um, we sample from this and we label using the teacher, and we scale it up pretty much 100x, uh, across different domains, languages, countries, where, uh, the LLM teacher model produces pretty good labels. We train the student model, and this is the model that actually gets served online. Um, and, uh, zooming into the student model, uh, also, this also has language models in it. Uh, but unlike the teacher model, it's not a cross-encoder. It, uh, is a bi-encoder, uh, which essentially means we don't have cross-interactions between the pin and the query, uh, representations. Uh, the pin gets embedded separately, query gets embedded separately, and it also uses a lot of other features like, um, SearchSage that we previously mentioned for both embedding the query and the pin. Uh, we have GraphSage embeddings, which Pinterest has published papers on, um, and OmniSage, and a lot of other embedding features for query and pin. But we also use, uh, a lot of pin-query text match statistics like BM25, which we've seen historically perform really well for predicting search relevance. Um, and the reason this scales well is the bi-encoder. Uh, bi-encoder large language models can scale really well, uh, when we, uh, use offline inference and caching. Uh, the pin embedding here is entirely offline and ferred on billions of pins. Uh, it uses predominantly the same text features that we mentioned on the teacher, uh, which helps distill efficiently. Um, and, uh, we only, uh, re-infer these embeddings every time that these inputs meaningfully change. Uh, meaning that, uh, every time that we, uh, need new embeddings, um, it's only going to run on a few sets of new pins. Um, and, uh, this is offline and ferred, so none of this is happening online when a user issues a search query. Uh, and the query embedding is pretty much, uh, real-time and ferred online. Uh, and search queries are pretty short. Um, they don't occupy too many tokens, which means, uh, we can keep the latencies for the query embedding up to, like, a few milliseconds. Um, and we also cache this, uh, because search queries get repeated a lot, and we get around an 85% cache hit rate. Um, and, yeah, this scales really well, uh, to actually serve Pinterest traffic. Um, the online results here, uh, the first four numbers are relevance, uh, measurements. NDCG, uh, precession at 8, uh, measured on the US, Germany, France, uh, specific segments that we zoomed into. Uh, we can actually see that we get relevance gains internationally, uh, even though we started with a very limited set of US data for this particular experiment. Um, and, uh, we also see that search fulfillment, which measures engagement on search, um, fulfilling actions, uh, also goes up. Uh, also on non-US, even though our, uh, starting data was predominantly US. And, uh, yeah, uh, large language models are very good at, uh, expanding across many different domains, countries, uh, even though, uh, they haven't been explicitly trained for this. ### Bonus Embeddings **Mukuntha Narayanan** [12:10] Um, and, uh, this is a bonus. Uh, we also found that relevance-tuned large language models produce really good, rich, uh, semantic representations, which are very good general purpose. Uh, this is the same production-relevance student model that I shared on the previous slide. Uh, and, uh, the pin embedding and the query embedding, uh, are basically free representations that we get from these models, uh, which can be used across Pinterest for representing pins and search queries. Um, we also use this to represent boards using the titles, etc. Um, and we found that using these embeddings, especially since they've been distilled from a large language model teacher and also have large language models in them, uh, they are very good at semantic content representations. Uh, and, yeah, they perform pretty well across, uh, related pins, home feed, and a lot of other surfaces where we've seen, uh, representations improve by adding these things. ### Recap **Mukuntha Narayanan** [13:14] Um, so let me go over the key takeaways again. Um, I think lesson 1, we found that LLMs are really good at relevance prediction. Uh, lesson 2, we found that visual language model captions are good, uh, ex-good ways to imbue them with, uh, image representations, and, uh, user actions are very good content annotations. Um, 3, uh, we found that knowledge distillation is a very good way to scale, uh, and efficiently serve models, uh, online. And, uh, lesson 4, uh, relevance tuning produces pretty rich representations, uh, that embed semantic representations for content really well. ### Q&A Thank you. Um, I wonder if there are any questions from the audience. Please come up to the mics. **Guest** [14:04] How did you decide which open-source LLMs to fine-tune? **Han Wang** [14:09] Yeah, that's a, yeah, that's a very good question. So we did a lot of experiments trying different language models. And, um, in the previous slide, we also shared some, um, performance for different language models. Yeah. So we did a lot of experiments and found the one that gives us the best performance. Yeah. **Guest 2** [14:30] Uh, if you could just walk us through somebody typing a search prompt. The confusion that I have is you have, like, LLMs, uh, building some sort of matching. Is it just being used for the label to be distilled, or how did you shim that into the bi-encoder? It wasn't really clear on the two-tier offline nearline and how the LLM search kind of influenced that. **Mukuntha Narayanan** [14:56] We used LLMs to distill into a student model which predicts search relevance specifically and produces five-scale relevance scores. Um, and it's served at the end of the search pipeline. It's, uh, the re-ranking stage. Um, like every recommendation system, we have a lot of, uh, CGs which are candidate generators. We have early-stage ranking, and this is one of the things that sits further down the pipeline, which actually predicts search relevance scores and, uh, is usedright before blending to actually produce a feed. So I think it's very similar to most recommender systems. **Guest 3** [15:34] So, uh, I have a, excuse me, uh, I have a question on how you evolved into this architecture. Like, I'm sure Pinterest has pre-LLM era search as well. Like, what limitations did you see in those systems that this new architecture solved for? **Han Wang** [15:56] So, um, if I'm understanding correctly your question, it's about what's the difference between the new system with the from the. **Guest 3** [16:03] Yeah. What was the, what was the driver to adopt adopting LLMs, uh, for, for in, in your search pipeline? Did it type, did it support new features, or is it, does it improve on the existing features where you had limitations? **Mukuntha Narayanan** [16:20] I think they definitely improve, uh, especially with visual language model captions. I think we were very effectively able to expand beyond limited markets for actually measuring relevance data and getting relevance data. And, yeah, these multilingual models are very good at, uh, getting synthetic data for different markets for, uh, them. **Guest 2** [16:48] Hey. Um, great, great talk. Um, I was wondering why, uh, or if, if the embedding model is inherently multimodal. Uh, because you have text, which is, uh, the, the query, and then you're matching against, um, either text, links, or images. And so how do you think about multimodality? **Mukuntha Narayanan** [17:07] It's definitely something we're exploring, but then, uh, on a lot of applications, I think we found, uh, visual captions are very good at ca uh, capturing what the image has. Um, and we have some very good captioning models in-house, which, uh, yeah, help us. **Guest 2** [17:24] Great. Thanks. **Guest** [17:29] Great talk. I, yeah, just a quick question. You mentioned that you saw improvements in other languages as well. Did you start with a common baseline model for all languages, or did you have to sort of, and just change the features for each language, or did you actually also start with separate models for individual languages? **Han Wang** [17:49] Yeah, that's a. **Guest** [17:50] Just curious how you actually saw the improvements manifest everywhere. **Han Wang** [17:54] Yeah. Um, so we, we used the, um, same model for all languages. And we because we u we are using the multilingual mo um, LLM, so we believe it can have transfer to other languages. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.