From MCP to Scale: Pipelines That Build Themselves — Rafael Levi, Bright Data

Alrighty, let's dive into it. So, uh, in the previous session—I don't know how many of you were here—but I talked about how MCP gives access to LLMs to websites that are behind capture, bot detection systems, and so on.

And in this session I want to talk about, um, how do you actually collect data on scales with LLMs. Because a lot of times I'm seeing on Reddit and other social media, it's like, "Oh, I need to scan 10,000 products, but that's so many tokens if I need to parse everything with LLMs."

So obviously you don't do that,right? So the whole session is: how do you build pipelines,right, with LLM. Instead of telling, "Hey LLM, can you go and, uh, parse this for me?" build a scraper that's going to parse it for me.

And I will demonstrate how easy it is, uh, with our skills,right? So Bright Data has skill sets that actually teaches your LLM on how to build a pipeline. With our MCP it can actually extract HTML so that it knows what are the selectors that it needs to parse, and so on.

So, um, the scrape tags,right? Write the scraper. Uh, I don't know how many of you actually wrote scrapers before. Okay, so you know the headache when all of a sudden things are missing, data is missing, you got to wake up.

I don't know if you guys did it for clients, and clients are like, "Oh my god, there's no—" So basically that's what it used to be. You write a scraper and you maintain it. As a matter of fact, sometimes you maintain it more than it takes you to write it,right?

So everything, especially if the website is constantly changing its selectors or if it's a React website, it gets really complicated,right? So an agent, it, it solves all that headache,right? It can explore with our MCP. It understands what the data is needed.

It writes the scraper and actually runs it and executes it and maintains it. As a matter of fact, I do collections on a daily basis and I have, every 30 minutes, I have an LLM that spools up, it checks what the data is collected, makes sure that everything is fine.

Everything is fine, it shuts down. If, for example, there's a, um, always set a validation for data,right? Let's say a data point is, uh, missing something. Your agent fixes it. Five minutes. You don't have to wake up in the middle of the night.

Uh, so let me demonstrate. So for this demo I want to use Claude Code. I don't know how many of you use Claude. If you're using Codex or whatever it is, I prefer Claude Code. I like how it's—I like it.

It does a great job. So the way that you build scrapers now, and it's ridiculous because, you know, back in the day it took weeks to set it up. We created a Bright Data like GitHub page,right? Bright Data skills.

Here, basically, your agent has everything that it might need, okay? Um, scrape builds, best practices, how to—everything. Everything that it needs in order to build a scraper. And what I want to show is, uh, first of all, let's—I'm going to start simple.

Go to

build me a scraper. And I wanted to record it because I thought it was going to take a long time, but now it takes literally like three minutes. Build me a scraper. Uh, two inputs: keyword search max pages for walmark.com.

Everybody's familiar with Walmart. Walmart has a very aggressive anti-bot system. So if anybody tried to scrape it, um, it just doesn't work. But through Bright Data.

The website name is wrong, I think. The Walmart thing might be a different one.

It will figure it out, but you're—you'reright. Let's just correct that. walmark.com. Um,

run a search for headphones, collect three pages.

Uh, so what it's going to doright now, it's going to go to our GitHub, it's going to get all the skill sets that it needs, how to build scrapers, then it's going to go, uh, with the MCP that's already connected to the Claude.

It's going to extract the HTML from the page. It's going to find all the selectors that it needs. And it's going to build you a scraper. And, uh,

soright now you can see a scraper as a markdown. It's basically extracting the, the, the text,right? So for the skills, we don't need the HTML, we just need the text. Uh, scraper as a markdown is a part of the MCP of Bright Data MCP.

Uh, the MCP has 5,000 requests for free. So if anybody wants to try it, you guys can try it for free. Um, you will need to just open an account with Bright Data, which doesn't cost you anything. And

let it do its thing. So

why is this better than having an LLM parse every single page, every HTML? It's—I'm, I'm literally afterwards I'm going to ask it to tell you how many, how much tokens it saves. So for the three pages, I've done this before.

It saves about a million tokens just from building the scraper, uh, and using a script to parse the HTML. And, um, even if you are not doing any scraping for production, but let's say that you want to find the best headphones for the money,right?

It used to be you go to Google, you go to some CNET where they, uh, compare everything. But here you can use the, the marketplace reviews. So I can tell it, like, "Hey, um, scan the five pages. Find me the best, uh, reviewed headphones."

So it could be even useful for you as a personal, in a personal way,right? So instead of it getting blocked, it can actually find you things that you need on websites that are behind Cloudflare. Oh, I forgot to delete the old one.

No, no, stop, stop, stop. Hold on a second. Delete the old scraper. Scr—I totally—I was testing it and I totally forgot to clean up.

Uh, let's do a different website. Pick a website. What's a popular marketplace in, uh, that's aggressive in blocking in UK?

Amazon. It's not that aggressive. You'll be surprised. I can scrape it with data center IPs. What's a popular website that everybody uses here?

Very, very. V-E-R-Y dot com.

V—very?

Very.

Like that?

Y.

Y dot com?

Yeah.

Oh, that's C-O-D-K? Okay, perfect. So this way I don't—I don't want to—stop it. New session.

Uh, uh, uh, let's split that.

Let's do that. Where's the new session? Oh, there it is.

Really?

What is wrong with my paste? Okay.

Uh, it's clothing store?

Uh, it's general. Headphones, you know.

Okay, so let's do headphones again. Same thing,right? So I don't know. I've never used this website. I've—clean test, no cheating.

Um, so, so what else does this give you? So again, market research. Um, for example, I was looking for a new apartment,right? I wanted to move a house. So with Claude Code, with Bright Data, I set up a listener.

Literally just telling, "Hey, listen, build me a scraper that will run every half an hour when in this area there's a, a house, private house under this price, notify me." That's all I did. In a few days I got a notification and now I live there.

So these things are not only useful on a scale where, you know, "Oh, I need to scrape millions of records," but even for your personal use. It's so easy these days. Claude Code does an amazing job,right? I don't know, uh, Codex does a great job as well, but I prefer Claude Code because, I don't know, it just, uh, gives me less headache.

Codex sometimes takes me on a wild goose chase. Um, how many of you Vibe Code?

It's, it's amazing. Why not,right? Anybody can build anything now. Um, so did you build it? I've got the structure, the search patterns. I don't know. We started a project. Let me build. Oh, okay. It's building a scraper. Um, this used to take, I don't know, if, if you're having a good day, maybe a full day, maybe a day and a half,right?

You need to go check out the selectors, figure it out. It used to be interesting and fun, but you don't need to do that anymore. So what I'm trying to show you guys is that with our MCP, with our infrastructure, we have over 150 million IPs.

Uh, with our unlocking technology where we even if it needs to run a remote browser. So if you want to, it can actually write you a browser automation. Uh, the browsers are running on our system. So I can literallyright now open a thousand browsers on this laptop that are running on our servers and they can go do things, whatever you need it to.

So it already got 90 products.

Okay, it just needs to fix the Unicode because it's in pounds.

Uh, almost done.

Can you also, uh, go into details about the MCP? Or is that something else?

Sure. I did it in a previous session, but while it's loading, let me tell you. So basically the MCP gives you, uh, agent 66 tools. Uh, some of them is basically, um, we have a system where it can send a curl to any URL and our system will literally get the HTML back.

It will salvage capture if needed and send it with the token. It will, uh, it knows exactly what headers and cookies the website needs. So basically it will make sure that the server thinks it's a browser and serve the HTML back.

So your agent can literally send curls, pull data without any questions. It can pull a full HTML. It can pull just a scrape as a markdown. Markdown, it's, it's to save tokens,right? You just want the text of the page.

You don't care about the HTML tags.

It doesn't like the pound.

Come on, you can do it.

Um, so that's number one. Second, we have about 500 different APIs prebuilt for different domains. So instead of actually getting a markdown, you can actually get a JSON of the product. For example, for Amazon we have prebuilt API.

You can listen, uh, when you add your agent, it can be like, "Okay, go and check on Amazon something." It doesn't even need to build a scraper. It can literally just send the, uh, the request and get the data back.

Um, on top of that, remote browser infrastructure and, uh, anything that needs, uh, anything that your agent needs to access the web, it has,right? So you don't get blocked. While this is running, I want to show you, for example, uh, tell me, "Why don't you Walmart do a search for headphones without MCP?"

So I'm telling it, "Go to Walmart, do a search for headphones and tell me what is the first result without the Bright Data MCP." Uh, it's going to do a fetch, which is going to get blocked.

Product security verification screen, robot or human, obviously,right? That's, that's the first thing.

Now do the same with Bright. Um, oh my God.

So now it's using scrape as a markdown. It's doing a search. It's only pulling the text. It's not pulling the HTML, just the text itself.

Always when you do a live, it slows down. The time is like.

It's too many people clicking a box on your.

Oh, actually it could be. It could be opening a browser and actually holding the button. You know, the Walmart has like one of those click and hold and it needs to hold it for like 30 seconds. So it could be doing thatright now.

So.

You brought a capture breaker.

We, we have built capture solving solutions. Like we literally have in-house. We have an AI solving capture.

Moving, clicking things and like, wow, it's perfect. So it already got

first result, blue headphone. Basically, there you go. It already has the results of the headphones without robot or human with full product list and name. Uh, so we here already have the output. Tell me if you were to do this manually, how much, how much more tokens have you used?

How many did you save? Let's just give it a breakdown. And this is probably the biggest issue,right? Tokens is expensive. I'm going through millions and millions of tokens a day. I used to go more, but I optimized it.

So, um, all of you are looking, how do we save money on LLM? How do we waste less tokens of web access? And, um, Bright Data has the solution. Instead of parsing the full HTML, create a scraper. Uh, instead of using, um, you know, okay, it pulls the HTML and it extracts the data, it builds the scr, uh, the parser.

But it's working very slowright now. I feel like everybody's Vibe Coding. So basically here's the breakdown,right? What do we have here? The big swings output, parsing 90 products,

input tokens, output tokens, total save. So it's about a 62% save of tokens. This is not a high number. This website, I guess, uh, it has, um, maybe a structured, um, HTML. I'm not sure.

But this is what I wanted to show you. And the best thing about it is that it will maintain it. If it breaks it, it will fix it. I can set up a loop,right? So for example, in Claude, I can set a schedule.

Every 30 minutes you do and run and check something. And this is what we are giving you guys, okay? With our MCP, your agent has access to all the web, uh, without getting blocked. No capture will stop it.

No robots, anti-bot systems will stop it. And, uh, you have no headaches of that. I don't know how many of you have, you guys probably dealt with blocking,right? When you build a scraper, it gets a little complicated. So this kind of removes all the headache for it.

Uh, any questions?

Where is the code?

Excuse me?

The code, the code to generate a scraper,right?

Oh, I can, if you, you want to see the code. I mean, it's here. We have the scraper. Let's check it out.

So it's using API web unlocker. This is the same thing as a scrape as a markdown with an MCP. It sends a simple request and gets the data back. This is the parser that it built.

I mean,

it looks pretty good. This is the, the schema that it built for the output. It's doing a keyword search. It has the max page,right? So I asked for two inputs, the keyword search and how many pages to scrape.

And hold on. Let's see. Let's see. We can run it through here as well. Where's the variables? API key zonkey.

Let's run it. Do another run on laptops. What are the two pages results?

And now you kind of control it. Like you don't even need to run it,right? It will trigger it for you. It will maintain it for you. Um, you can create a script that will automatically do all of this.

I prefer the visual interface for the demonstration. Sometimes, you know, like looking at our code, it's a little more complicated. Butright now, basically I can ask it any questions. It will actually run the script. So it's now wasting tokens.

To execute the script, maybe like 60 tokens,right? Right now. So it's going to literally get all this data for 100 tokens and then you can do whatever. You can ask it questions about it. But now we're working with a JSON.

A JSON is much more token efficient than any markdown or any other file,right?

It's all of the usual.

Can I, can you, can you do this on all with authorization on the site as well?

No, we only talk, we only deal with public data. So behind login, it's, um, it's private data. And, and, uh, just, you know, just a heads up. If you create an account, you accept terms and conditions of the website.

Always check terms and conditions of the website. If they say don't scrape, don't use robots, and you do, then that company can actually sue you. And this is happening. You probably heard it in the news all the time.

LinkedIn is suing them. Everybody's suing each other because they, the dataright now is like the new gold, the oil,right? Everybody's trying like, "Oh, this is my data." Elon Musk took over Twitter, locked it down. That's it. Literally like, uh, it used to be so open and now it's all locked in.

A few accounts that you can actually scrape. So we're seeing this on every single level.

Very slow today. Um, so yes, also same thing. If you accept, uh, there's like, uh, terms and like if you do a search and there's a checkpoint, I accept terms and conditions. Always be careful with that. So we only deal with public data.

So nothing behind login. We don't accept terms and conditions. And we actually won a few lawsuits. We were sued by Mellon. We were sued by Elon Musk a month after he took over. And, uh, the judge said it's very simple.

Public data is public data. It doesn't matter how you collect it. It doesn't matter what you do with it. It's public. You know, it's like walking on the street. You write down the prices on the counter and then you sell it to somebody.

It's public data. It's available. You can do whatever you want with it. Okay. So

as usual, it gets stuck on a live demo, but

we can, we can run this manually.

Oh my God.

What the, oh, it's setting a variable. It's doing this. No, it's not going to work.

It needs to run that batch in the log.

What am I missing?

Still stuck? Oh, okay. It's done. Okay. So here we got. We got, let's see, how many tokens? How many tokens did you spend on getting these?

So even if you're not building pipelines for any, um, you know, big companies, then you don't need to scale. For personal use, I always have the MCP connected. I don't ask the, you know, LLM to go do. I always ask, build a script that can later on be used by it.

It's using its own script to save the tokens.

Literally like we're talking about 1,000 tokens where if, uh, you know, if, if it needs to go through the JSON, it's like 10,000 tokens. We're talking about like literally pennies compared to what it would be to actually scrape it.

Any more questions? How about this? Anybody has problems with actually getting data?

No, no question. Everybody is like, "I'm not sharing my problems. I have no problems." And then they go home and like, "Oh, I do have a problem." But in any case, listen, we have a booth. If you don't, if you want to talk in private and you have questions about data or access of your LLM, feel free.

I'm always willing to help. Um, we can connect on LinkedIn, by the way, guys, if you want. Let me just summarize a few things what we did here. So, um, self-healing pipeline. When you have, when your agent actually has access to a blocked website and you're working, I mean, the MCP is mostly useful in about 20% of the, of the domains,right?

The ones that have Akamai, Datadog, Cloudflare, those heavy protected domains, they're the usually the most juiciest one,right? Like real estate or, uh, big e-commerce places. Uh, with our MCP, it has access to it. If it has access to it, it can build a scraper.

If it can build a scraper, it can maintain the scraper. This is basically the whole thing that I wanted to show you, that in Bright Data, we have all the tools that your agent might need to explore, to build and maintain, uh, a pipeline,right?

So if you want to set up a listener for an apartment and you're looking to move into a cheaper place, or maybe you want to book a table in a restaurant where it's always packed, and literally I, I, I have a listenerright now waiting for two months already.

Everybody's like bookingright away. So as soon as this spot opens, it will automatically book a spot for me. It can be very useful in just kind of like even, even the small things, personal things,right? I'm not talking about just enterprise scale where I need to download a million records.

Okay? But having access to all the websites, it's, it's, it's important.

So apart from scraping, can you also do actions on the website?

Yes, of course.

Fill a form.

Fill up a form, submit it as well. Yes. The only thing you can't do is log in,right? So if you have, um, let's say you need to perform a search,right? You cannot generate the URL. You need to click buttons.

Let's say flights. You want to check flights availability, Skyscanner or something like that,right? The URL is usually a hash and you can't, uh, do anything with it. So yes, LLM can spool a browser, remote browser. Even if it's a geo-restricted site, it can be like, "Okay, I want IP from the United States."

So it's open a browser with the United States IP and then it goes and clicks things, fills it up and so on. The beauty of it is that our browser will mimic real human behavior. So when your agent clicks, it's not a teleportation.

There's a mouse prerecorded, like a real human being moving it. When it types, it will type a little slower, speed up, like maybe even mistake, so on. So we have prerecorded typing. We have prerecorded mouse movements. So if the website has a tracker that's constantly sending to the server what the user is doing, it will look like a real human being.

Doesn't matter what your agent, even if it's a low like, um, for browsing agents, I'm, I've never used the top models,right? For example, if with Claude, Haiku model is more than enough for browsing. If it's being masked that it's a real human, it works just fine.

Um, so that's pretty much it. If you guys want to connect on LinkedIn, feel free. I'm always willing to help. If you have any questions regarding how to get data, if you have problems with accessing anything, feel free to message me.

I feel like there's been so much more than 15 minutes. I love it. It's the second time I'm doing a speech and I got like half an hour instead of 15 minutes. It's amazing. Um, now's the time if you have any questions, discussions.

No, nothing. Great. Okay, guys, I guess I will conclude this session on this. Uh, feel free to come to the booth on the third floor if you have more questions and, um, let's keep the public data public. Thank you.