Someone on r/quant recently asked how pods and macro shops source web-scraped alternative data, how they decide between building and buying, and how LLMs are impacting that decision. My answer became the basis for this blog post.

Let's say you're a PM and find a clear alpha signal that requires web scraping. You can now just pull up Claude Code and generate a scraper that runs on a schedule on your machine. But a valuable dataset rarely depends on a simple one-off scrape, and as you run your script daily, you quickly run into anti-bot defenses, layout changes, compliance, and infrastructure problems (or the script might even accidentally wipe your hard drive). Eventually, you turn to your central data team to ask if they could take your vibe-coded scraper, deploy it reliably, monitor the output, and maintain it whenever it breaks.

We hear versions of this story frequently these days. After years of building web scraping pipelines for hedge funds, here’s what we see firms doing and what we've learned from working with them on these problems.

Creating simple scrapes got cheap. Maintenance did not.

Tools like Claude Code and Codex can generate a working scraper prototype in minutes. Coding models have dramatically reduced the cost of getting started, but they have not eliminated the recurring cost of running a reliable data pipeline.

Maintenance is almost always the silent killer in the build-versus-buy debate. A scraper can work perfectly at first, but then a website changes, proxy costs spike, or the output schema begins to drift, and you end up wasting half a developer's time just keeping the pipeline alive.

Most large firms with serious engineering resources have in-house web scraping infrastructure and teams. These are often central data teams that manage and maintain scrapes for different teams and pods.

Small-to-mid-sized firms often outsource because they don't have the manpower to build, and especially maintain, a large fleet of pipelines. They often still follow a traditional and slow process for sourcing web data, going through an internal Jira ticket, compliance review, data engineer, web scraping provider, and QA.

Meanwhile, central data teams are trying to move away from siloed datasets to a centralized, AI-ready data foundation where public, vendor, and internal data is easily accessible to analysts and AI systems. A growing and distributed collection of unmanaged scrapers works directly against that goal.

Firms that try to build all their web scrapers in-house tend to struggle with four challenges:

• Blocking is an increasing issue, and most funds don't have the sophisticated proxy and unblocking infrastructure to keep up with it.
• Operating web scrapers at scale is hard because every pipeline adds maintenance and technical debt. At scale, you need proper observability, infrastructure monitoring, alerting, and incident management. Only a handful of specialized firms can do all of that in-house.
• Ensuring provably correct data (precision and recall) is hard, and as soon as you add LLMs to the game, you're dealing with non-deterministic behavior.
• Governance gets more complicated because compliance may want to audit or approve scrapes, while PMs and analysts now vibe-code their own scripts with Claude Code. Without a managed process, this creates security, compliance, and data governance risks.

Build, buy, or both

The answer to this, as with every complex engineering question is "it depends".

If the data is standardized and widely available (e.g. filings), firms tend to buy a pre-cleaned feed from a provider that handles all the scraping and structuring. Buying off-the-shelf datasets always embeds the provider's opinions in it though, and you might not agree with it.

A reason to build in-house is when the scraping logic itself is the core IP, or if the source is so niche and proprietary that no vendor can touch it. Even then, they often prefer to buy raw feeds from providers and handle the structured parsing themselves because managing anti-bot bypasses at scale is a full-time engineering problem on its own.

Kadoa’s Web Scraping OS supports the entire spectrum: firms can use it to collect raw web data while keeping proprietary processing in-house, run the complete workflow end to end on Kadoa, or deploy on-prem when sensitive IP must remain within their environment. It gives engineers and analysts a single AI-native platform to build, monitor, and ingest public data.

Firms want to keep the alpha-generating parts of their data infrastructure in-house, and avoid owning commodity infrastructure unless they can operate it well at scale. The first vibe-coded data pipeline is cheap. Operating hundreds of them is very expensive.

If you’re working through the same decision, feel free to reach out.

Adrian Krebs

Co-Founder & CEO of Kadoa