At Owner, one of the main ways we help business owners is by showing them how the digital world sees them. This is the first step of many to help them be more competitive, stand out online, and attract more customers. For a local business, a big component of their online presence is their discoverability through search engines and therefore their websites.

In order to see what your website looks like and how it’s structured, we use web scraping. We did not build this in-house but we use various third-party providers to faciliate that task. One of the issues that plagued us for a long time was how unrealiable the web scraping task in our Grader tool could be.

If you went to the grader.owner.com last year and entered your restaurant, there was at 82% chance we would successfully be able to access and analyze the HTML on your site. This means roughly 1 in 5 attempts could not give restaurant owners a thorough assessment because we lacked complete information.

We would either get a web page quickly, or requests would time out. Our latency distribution clustered around two points, forming a bimodal distribution.

Distribution of latency on web scraping attempts

That is unacceptable. Something had to be done, so I set out on this mission.

How would I accomplish this?

The first thing I did is simply shop around for a different vendor. I tried many of the most popular ones. Heck, I even went on the second page of Google search results for goodness sake!

No man’s land of desperate people

None solved my problem… Until I understood why.

As it turns out, web scraping inherently follows a bimodal distribution mode regardless of the vendor. Therefore, the trick to raise performance is to use active parallel redundancy to compound the failure-probability (\(\mathcal{q}\)) of the individual attempts to near-zero for the whole. (\(P(X \geq 1)=1-q^n\))

In the rest of this article, I will ellaborate on the details of what this means and how the algorithm came together that got us to a 99% success rate.

How Web Scraping Works

On Paper

Now if you’ve done enough interviews, you have probably seen this question before. This is a popular one to ask in the system design stage: Design a web scraper.

Simple web scraper design

At a high-level you have a Collector, Scraper, and Storer.

• The Collector finds the URLs we want to crawl and adds them to a queue.
• The Scraper picks up the task and attempts to perform it.
• On success, the raw HTML is stored in Blob storage, otherwise it goes back into the queue to be retried up to \(\mathcal{n}\) times before moving it off to a DLQ (Dead Letter Queue).

This is the textbook example we talk about in classrooms and the kind that get you a “Hire” rating on your FAANG internship interview.

In the Real World

Now in the real world, web scraping looks quite different.

Realistic web scraper design

The reason this diagram is very busy is because modifications have been made to deal with:

• this niche language called Javascript
• CDNs and WAF

A large amounts of sites nowadays are SPAs (Single Page Applications) powered by Javascript so naive curl requests would leave you blind to most of what people see, thus we need simulated browsers in a VM. Another heavy influence is many domains use CDNs & WAF to block out excessive traffic from repeated datacenter IP addresses as anti-bot measures. Therefore, the need for a commercial and residency proxy pool toS rotate through is a table-stakes.

Understanding this complex supply chain helps clarify why our distribution chart looked like this:

Distribution of latency on web scraping attempts

A given scraping request \(R\) could either hit all healthy nodes along the path (from the VM, to the Chromium process, to the egress proxy gateway) and complete quickly or it would not and get stuck until it timed out at 30 seconds.

This is the very nature of web scraping! It is also why none of my provider-switching worked.

It’s not that some vendors were bad and others weren’t (though they are not all equal to be very clear), it’s just this is the name of the game for all players involved. This is what contributes to latency clustering and the observed bimodal distribution.

Once it was clear this is inheritent to the task, the approach to overcoming the challenge had to take that into account.

Mathematical Intuition for Compensating

I mentioned earlier that all providers are not equal, and we will dig into that later. But for now, let’s assume they are and the success rate across the board is 65% for each.

Pop Quiz:

If the gas station down your street sold scratch-off ticket and each one had a 65% chance of a huge payout, what is the minimum number of tickets you should buy to ensure you have a 99% chance of winning?

How about 99.99% ?

For the sake of our example, let us define:

With one ticket, the probability of winning is simply:

The useful question is not “what is the probability that every scratch-off is a jackpot?” It is:

For \(n\) independent attempts, the chance that all of them fail is:

Therefore, the chance that at least one succeeds is the complement of the two:

More generally, if \(X\) is the number of successful calls, then \(X\) follows a binomial distribution \(\operatorname{Binomial}(n,p)\) where \(n\) = attempts and \(p\) = success probability. It is formally defined as:

For our purpose, we care about one or more successes:

To find the minimum number of attempts required for a desired confidence level \(c\), solve the inequality:

Since we cannot buy a fraction of a ticket, we round up:

With \(p = 0.65\) this means for targeting a 99% success probability:

Checking the result:

So 5 independent trials (or tickets) give us approximately \(99.47\%\) confidence that at least one is our ticket to our dream mansion in Lake Como.

Taking this back into the context of a web scraping request: What about a \(99.99\%\) success probability?

So 9 independent trials give us approximately \(99.99\%\) confidence of at least one successful response.

In our system, this means we can compensate for an unreliable single scrape request by making many of them. Assuming each provider has a \(65\%\) independent chance of success, five attempts get us beyond \(99\%\) reliability, while nine attempts get us beyond \(99.99\%\).

Where Rubber Meets Road

Now that a direction has been established, let’s look at how we could implement this:

const pool = [
  ScrapeFast(),
  BlazingScape(),
  PremiumScraperBlue(),
  Crawly(),
  WebCrawlPeach(),
  // etc...
]

/** Attempts to scrape a url by leveraging our Scraping Pool */
function scrapePage(url: string): string | null {
  for (provider in pool) {
    try {
      const res = await provider.scrape(url);
      return res.html;
    } catch (err) {
      console.error("Failed to scrape", { err })
    }
  }

  console.error(`All providers failed to scrape url: ${url}`)
  return null;
}

This should mostly work, but do you see anything wrong with this code?

.

.

.

     ...space to think...

.

.

.

1. We walks the vendor list sequentially

Remember that our failure mode is an attempt that got stuck and runs the full wall-clock time until it times out. Thus, the worst case scenario would mean hitting this failure mode on all five sequential calls which would have users waiting way too long to get a result.

Where \( \tau \) is the timeout ceiling and \( k \) the number of attempts. Simplified as \( k\tau \).

⚠️ This is a real edge-case because there is a surprisingly large number of business owners who point their Google maps profiles to websites that are no longer actives. Often times, the domain has expired or the template is misconfigured. Those scrapping attempts will never succeed and we want to surface that insight quickly.

This guarantees success at the fastest speed for our customers. It does come at the trade-off of \(k\) increased cost on a per-attempt basis but there are ways to play around with the slider intelligently and this small increase pales in comparaison to the additional value we create for our customers in exchange.

2. We assume vendor success rate is static

Our data shows this is not the case.

Below are charts for two of our vendors for the past seven days (as of Oct 12, 2026).

While some are generally better than others, their individual reliability flunctuates. One can have a total outage on a random Saturday at 7am. Another that is consistent at ~80% can suddenly crash below 40% for a couple hours.

What’s truly needed is a dynamic algorithm; one that has a short memory and re-evaluates all the vendors in the pool in real time to decide which ones to use right now based on recent performance.

Bonus

This real-time re-ranking allows us to maximize a multiplicity of variables. For instance, you might wonder:

“Why would they keep the purple vendor in the pool when it appears to be underperforming?”

Cost is an important axis. If purple is 2x cheaper than blue, then all else being equal, we should use it where we can.

Corrected Code

That is what was built.

1. For the first 15 minutes, the algoritm bootstraps itself by picking at semi-random which providers to include in its attempts budget.
2. For each vendor, We store in Redis how long the request took and whether we can consider it a success or not.
3. Once enough data is collected, we score the vendors with a score = 0.60 * stability + 0.25 * latency + 0.15 * cost function and sort by highest score first
4. We take the top \(N\) (in our case 3) providers from the pool and we concurrently race them
5. We return the fastest result to the caller while we let the other two finish so we can track their completion time.
6. We sample \(S\) percent of requests (in our case 5%) to fire off the whole pool and not just the top 3 so we can keep the scores fresh and current for the rest of the pool. In case the top 3 degrade, we know who to promote.

Here is a high-level snippet for what that looks like:

// Representative code only. Production has retries, tracing, and safeguards omitted here for clarity.
const PROVIDERS = [
  { id: "blue", costPerRequest: 0.012, scrape: (url) => blueProvider.scrape(url) },
  { id: "purple", costPerRequest: 0.006, scrape: (url) => purpleProvider.scrape(url) },
  { id: "orange", costPerRequest: 0.018, scrape: (url) => orangeProvider.scrape(url) },
  { id: "green", costPerRequest: 0.009, scrape: (url) => greenProvider.scrape(url) },
  { id: "red", costPerRequest: 0.015, scrape: (url) => redProvider.scrape(url) },
];

// These are all dynamic feature flags in production
const ATTEMPT_BUDGET = 3;
const EXPLORATION_RATE = 0.05;
const TIMEOUT_MS = 30_000;

const MOST_EXPENSIVE_PROVIDER = Math.max(
  ...PROVIDERS.map((provider) => provider.costPerRequest),
);

export async function scrapePage(url: string): string {
  const providers = await chooseProvidersBasedOnScore();
  const attempts = providers.map((provider) => attempt(provider, url));

  // All requests start immediately. Promise.any resolves with the first
  // successful result, rather than waiting for every provider to finish.
  const telemetry = Promise.allSettled(attempts);

  try {
    const winner = await Promise.any(attempts);
    return winner.html;
  } catch {
    throw new Error("Every scraping provider failed");
  } finally {
    void telemetry;
  }
}

async function chooseProvidersBasedOnScore() {
  const outcomesByProvider = Object.fromEntries(
    await Promise.all(
      PROVIDERS.map(async (provider) => [
        provider.id,
        await getRecentOutcomesFromRedis(provider.id),
      ]),
    ),
  );

  // Occasionally race the whole pool, so every provider continues to earn
  // fresh measurements and can work its way back into the top three.
  if (Math.random() < EXPLORATION_RATE) return PROVIDERS;

  return [...PROVIDERS]
    .map((provider) => ({
      provider,
      score: providerScore(provider, outcomesByProvider[provider.id]),
    }))
    .sort((left, right) => right.score - left.score)
    .slice(0, ATTEMPT_BUDGET)
    .map(({ provider }) => provider);
}

function providerScore(provider, outcomes) {
  const successes = outcomes.filter((outcome) => outcome.success);
  const stability = successes.length / outcomes.length;
  const medianLatency = median(successes.map((outcome) => outcome.latencyMs));

  // Every component is normalized to [0, 1], where 1 is best.
  const latency = 1 - clamp((medianLatency ?? TIMEOUT_MS) / TIMEOUT_MS);
  const cost = 1 - clamp(provider.costPerRequest / MOST_EXPENSIVE_PROVIDER);

  return 0.60 * stability + 0.25 * latency + 0.15 * cost;
}

async function attempt(provider, url) {
  const startedAt = performance.now();
  let success = false;

  try {
    const response = await provider.scrape(url, { timeout: TIMEOUT_MS })
    if (!response?.html) throw new Error("Provider returned no HTML");

    success = true;
    return response;
  } finally {
    const latencyMs = Math.round(performance.now() - startedAt);

    // Telemetry must not delay the user-visible winning response.
    void recordOutcomeToRedis(provider.id, { success, latencyMs });
  }
}

This is a pure meritocracy, where only top performing vendors survive.

Conclusion

Today HTML web scraping is essentially a solved problem at Owner.

Going deep on the problem and understanding clearly how modern web scraping works at scale clarified where the bottleneck was. It was not any specific vendor, it was the task-space.

Thus, by designing a system where we have active redundancy and we smartly reallocate where compute budget goes (through proxy of external vendors) in real-time based on a multi-variable scoring function, we were able to take an inheritenly flaky task and make it incredibly robust and consistent while reducing user-perceived latency.

Many valuable lessons were learned that were transposed to some of our other systems to make them more reliable.

@article{kapuku2026scraping,
  title = {The Bimodal Nature of Web Scraping & Techniques for Engineering Reliability},
  author = {Verdi Kapuku},
  year = {2026},
  month = {September},
  url = {https://verdikapuku.com/posts/bimonal-nature-of-web-scraping/}
}