Building LinkedIn's GenAI Platform — Xiaofeng Wang

Uh, it's my pleasure here to share our journey on building out LinkedIn's GenAI platform. Uh, my name is Xiaofeng Wang, I'm the manager of GenAI Foundation.

Uh, let me try it one more time. Cool. Uh, in today's talk, I'd like to first share our journey on building out this platform, especially on why we're building it, how we build it, and what we're building it.

After that, we will talk about, uh, some thought process on why this platform is critical for today's agent world. Uh, hopefully after that you agree with me this is a critical component in your component, uh, in your company, and you also want to build this team.

I want to share some tips on how to build such a team, how to hire for such a team. Towards the end, we will share some key takeaways and lessons learned. Before we dive into this application, uh, platform journey, I think it's important to first talk about the GenAI product experience, because that's essentially what our platform is supporting for.

Back in 2023, LinkedIn launched the first, uh, formal GenAI feature called Collaborative Articles. This is a kind of straightforward, uh, GenAI feature if we are thinking in today's standard, because it's a very simple prompt-in, string-out type of application.

Uh, we leverage ChatGPT, uh, I mean, we leverage GPT-4 model, uh, to create the long content, uh, articles on the platform and then invite our members to comments on it. At this stage, our team helped to build some key component behind the scene, including the gateway to centralize the access to the model, uh, some Python notebook for the prompt engineering.

Uh, but at this time, we actually have two different tech stacks, uh, to serve the, uh, experience. In the online phase, we use Java, and in the backend, we use Python. Uh, we wouldn't call this as a platform at this time.

Very soon we realized, uh, there are some limitations for this simple approach, especially it lacks the capability to inject our rich data into the product experience. Then in mid-2023, we started to develop the second generation of the GenAI product, uh, internally we call it Copilot or Coach.

Here we're showing one popular such experience on LinkedInright now. Uh, basically it looks at, uh, your profile and the job description, and then, uh, use, uh, some RAG process to give you personalized recommendation on if you're a good fit to the job.

At this time, we started to build, uh, some platform capability. Uh, specifically in the center of our platform, we built the, uh, Python SDK on top of the popular, uh, LangChain framework to orchestrate LLM calls. And it also provides the key value to integrate with our large-scale infrastructure, uh, in this SDK, so our developers can easily assemble an application.

We started to unify the tech stack at this stage because we realized it's really costly to transfer the Python prompt into the Java world, not to mention the error during this process. We started to invest on the prompt management or prompt source of truth.

This is a sub-module at this stage, uh, to help developers to version their prompt and to provide some structure around their meta prompt. Uh, the most important piece I'd like to call out here is conversational memory. Uh, this is, uh, infrastructure to help to keep track of the LLM interactions and the retrieval content, and then inject those content into the final product.

It will help us to build this kind of conversational, uh, bot.

Now, uh, zooming into this year, uh, actually in the last year, uh, we launched our first ever, uh, real multi-agent system called, uh, LinkedIn Hire Assistant. Uh, this is, uh, multi-agent systems to help our recruiters to do their work, uh, efficiently.

Especially it automates several tedious tasks normally recruiters need to do manually, like, um, uh, post the job, um, and evaluate hundreds of candidates, then, uh, reach out to them.

Our platform also started to evolve into the agent platform. Uh, from the framework side, we extend the support of the Python SDK into a more large-scaled, uh, distributed agent orchestration layer. It will handle the distributed agent execution and also handle the more complicated scenarios like retry logic and traffic shift.

Uh, for folks who build agent, uh, I think you probably know the skills or APIs are one key aspect of the agent because we expect, uh, this agent to perform some action. One investment we did at this, uh, time is around the skill registry.

Basically we have a set of tools, uh, to help our developers to publish their API into this centralized skill registry. This skill registry can handle the skill discovery problem, skill invocation problem, so in your application it's actually very easy to call the API to perform some task.

Another key component, uh, we invested at this stage is on the memory. In addition to the conversational memory, we extend, uh, its capability into the experiential memory. Essentially it's, uh, memory storage to, uh, extract and analyze and infer the contextual knowledge from the interaction between the agent and our user.

We also organize this memory into different layers, including the, um, uh, working memory, long-term memory, collective memories. Uh, this can help our agent to be aware of the surrounding content. Uh, lastly at this, uh, time we also realized the observability is super important because agent, uh, one key aspect to define agent is autonomies,right?

Uh, because agent can decide what API they can call, uh, what LLM they need to call. So it's actually very hard to predict its behavior. So we started to invest on the observability. Uh, particularly we built our in-house solution on top of the OTel to keep track of very low-level granularity of the, uh, telemetry data.

So we can use this data to replay the, uh, agent call. And we also add, uh, extra layer of the analytics on top of it, so we can use that to guide the future optimization of our agent systems.

Let's put together all the components we built for this platform. Uh, we can classify them into four layers basically, including the orchestration, prompt engineering, tools and skills invocation, content, and the memory, uh, management. Uh, of course that's not everything in the LinkedIn GenAI ecosystem.

Uh, in addition, we have our sister teams to build out the modeling layer like fine-tune the open-source models, responsible AI layers to make sure the agent is behaving according to our policy and standard, and also the, uh, AI platform or machine learning infrastructure team to host those models.

The key value proposition for this, uh, GenAI platform is actually to, uh, be the unified interface for this complex ecosystem. So our developers don't need to necessarily understand all those individual bugs when they build, uh, their application. Instead, they can leverage our platform to quickly access this entire ecosystem.

Uh, for example, uh, in our SDK, the developer can just switch one parameter in the one line of the code to switch from the OpenAI model to our on-prem model. Of course they still need to do the prompt engineering, but that reduces a lot of the complexity on the infrastructure integration phase.

Uh, last but the most important is because of this is a centralized platform, uh, it provides a place to enforce the best practice and governance. So we can make sure our developers are building the applications efficiently but also responsibly.

As you can see from our journey, we actually started to build this, uh, platform piece by piece, and then this platform started to emerge. If we take one step back and think, uh, do we really need this platform at this time?

Especially there are lots of, uh, um, uh, vendor product, uh, on this space. Shall we buy it, build it, and why do we need to buy it or build it? Uh, here are some thoughts. Um, the short answer is yes.

The reason behind it is, uh, we feel like, uh, GenAI is totally different than new AI systems compared to the traditional AI systems. So in the traditional AI systems, there's a clear cutoff between the, uh, AI model optimization phase and the model serving phase.

So AI engineers and the product engineers can operate in two different tech stacks. Uh, they usually don't, uh, need to, uh, work on the same code base. But in the GenAI systems, what we're seeing is this line between the optimization phase and the serving phase disappeared.

Basically everyone is an AI engineer who can optimize the overall system performance. This actually created a new challenge of the tooling and the best practice in the company. Essentially we think these GenAI systems or, uh, agent systems is a compound AI system.

Here we borrow the definition from Berkeley AI Research Lab. Uh, compound AI system can be defined as a system which tackles AI tasks using multiple interacting components, including multiple calls to model, retrievers, or external tools. As you can see, this is actually a skill across AI engineer and product engineer.

And I believe this, uh, GenAI platform is trying to bridge this gap.

To summarize, uh, we believe this platform is critical for your success, mainly because it can bridge the skill gaps between those two groups of engineers.

Okay. Let's say if you want to build this, uh, platform in your company and how to hire it is a frequent question, uh, I heard. Um, I basically look into, uh, my great engineering team and, uh, extract all the key qualifiers from those top engineers.

And, uh, I put all the qualifications here. Uh, the ideal candidate in this team is a strong software engineer, uh, who can build infrastructure integration. They have a good developer, uh, PM skills to design the interface. Uh, ideally they have the AI and the data science background to understand the latest techniques.

They are the people who can learn from the latest techniques, but at the same time they are hands-on. Unfortunately, it's really hard to guide those candidates. If you guide them, uh, it's probably worth more than a unicorn. Realistically, we are making multiple trade-offs in the hiring.

Uh, here are some principle, uh, we follow and it's actually working pretty well. Want to share here.

In terms of the core skills, uh, we usually prioritize the stronger software engineer skills over the AI expertise. This might be controversial, but, uh, uh, uh, we can discuss if you're interested. Second is instead of hiring for experience or degrees, we hire for the potential because this field is evolving so fast.

Most of the experience might be outdated. In case you won't be able to find a single engineer with all the qualifications we are showing here, uh, the way we are solving this problem is to hire a diversified team.

So, so for example, uh, in our team we have some full-stack software engineers, we have data scientists, we have AI engineers and data engineers. We also have, uh, fresh grads, uh, from the top research university and also, uh, some people from the startup background.

And then we put them together, uh, into the project. What we've seen is based on those collaboration, those strong engineers start to pick up new skills in the project. And very soon they started to grow into these ideal candidates.

Uh, lastly is, uh, want to emphasize is, uh, the critical thinking. Uh, one constant topic, uh, in our team meeting is, uh, no matter what we're buildingright now, it will be outdated within a year or even less than six months.

So we consistently evaluate the latest open-source package, talking with vendors, and deprecate our solution more proactively.

Cool. Let's talk about, say, uh, some, uh, key takeaways, uh, especially on the tech stack choice. If possible, we strongly recommend Python. We started with Java and Python. Uh, there are some back and forth of the debate internally, but finally we picked Python.

And I think that's theright choice, especially most research and open-source, uh, are in this space. Based on our experience, it's also scalable. In terms of the, uh, key components you want to build in this platform, the first one is a prompt source of truth.

Prompt in some way is like a traditional model parameters. You want to have a really robust systems to version control your prompt. This is really, really critical for the operational stability. You don't want to accidentally edit your prompt in production and, uh, see some really side effect.

Second key component is on the memory. I think in today's, uh, meeting, uh, uh, I mean today's talk, some already talked about it. Memory is a really key component to inject your rich data into the agent experience. Lastly, in the agent era, uh, one key new component we are building is on the uplifting our APIs into skills which can be called from the agent easily.

So you can, uh, build some surrounding tooling and infrastructure to support this need.

Allright. Let's talk about how to, uh, scale this solution and got guided adopted. Uh, from our experience, instead of trying to build this full-fledged platform at the beginning, try to solve immediate need. For example, we started with a simple Python library to support orchestration.

Then we started to grow into all the components we're seeing here. Second is, uh, focus on the infrastructure and the scalable solution. At LinkedIn we actually have a pretty good success story by leveraging our, uh, messaging infrastructure, uh, to be as a memory layer.

Uh, it's both cost efficient and scalable. Lastly is, uh, focus on the developer experience. By the end of the day, this platform is trying to help developers to be as productive as possible. Their adoption is a key for the success.

If you can design this platform, please focus on, uh, how to align your technology with their existing, uh, workflow so it will ease adoption and, uh, be more successful. Uh, we actually have lots of low-level details on the technical side.

Uh, if you are interested, please check out our engineering blog post on LinkedIn, uh, by Karthik, Sandeep, and myself. Uh, with that, uh, thank you for your attention. And, uh, if you are having more questions, happy to answer that after the talk.

Thank you.