The Web Was Built to Be Read. AI Is Learning to Consume It.
AI search promises better answers, but its appetite for information is creating costs that most users never see.A research agent receives a question and divides it among four workers. Each worker runs a web search, finds the same technical report, and downloads it. A verification step then…
AI search promises better answers, but its appetite for information is creating costs that most users never see.A research agent receives a question and divides it among four workers. Each worker runs a web search, finds the same technical report, and downloads it. A verification step then retrieves the report once more because it cannot see what the other workers have already read.The application has paid for five retrievals. The publisher has served the same document five times. The answer contains only one piece of evidence.That small example describes a broader infrastructure problem. As AI agents take on longer tasks and use more parallel workers, inefficient retrieval becomes expensive for both the application and the websites supplying its information. It can also make the final answer less reliable.A responsible retrieval layer needs to do more than fetch pages. It should coordinate access, preserve evidence, manage freshness, and make failures visible.The Cost Exists Outside Your ApplicationOn August 29, kernel.org administrator Konstantin Ryabitsev published a detailed account of automated traffic hitting the site’s infrastructure. Across five geographically distributed servers with 90 CPU cores, he reported that 14–16 cores were continuously occupied rendering Git commit pages for scrapers.The repositories were already available for cloning. Instead, automated clients requested individual HTML pages and repeatedly collected overlapping content across repository forks.Kernel.org remained responsive, so this was not an outage story. It was an account of persistent infrastructure waste. Ryabitsev also presented the attribution to AI training as an operator’s assessment rather than a complete measurement of every request. His original post, “Creepy crawlies”, is worth reading in full.For an agent developer, the lesson is straightforward: a request that looks cheap in your dashboard can be costly at the source. Dynamic pages may trigger database queries, rendering, and other server-side work before returning a relatively small amount of text.Ten retrieved URLs can also represent one underlying document. Counting successful requests tells you little about the amount of distinct evidence obtained.Model Retrieval as a Shared LayerIt is tempting to let every agent call search and page-reading tools directly. That is convenient in a prototype, but it gives each worker its own view of the web.A production system benefits from a shared retrieval layer between agents and external sources. That layer should handle five related jobs: discovering possible sources, selecting which ones deserve deeper reading, coordinating fetches, storing evidence state, and reporting the result back to the agent.Its purpose is narrow: obtain sufficient evidence without repeating work already done.A source record might look like this:type SourceRecord = { canonicalUrl: string; retrievedAt: string; publishedAt?: string; contentHash?: string; status: "available" | "blocked" | "failed" | "stale"; evidenceGroup?: string; etag?: string; lastModified?: string;};The canonical URL prevents trivial URL variations from creating separate entries. The content hash helps detect copies or revisions. The status records whether a source was actually inspected. The evidence group connects multiple URLs that repeat the same underlying report.Keep retrievedAt and publishedAt separate. A page fetched this morning may contain information published six months ago. Retrieval freshness and information freshness are different properties.Collapse Duplicate Fetches Before They Leave the SystemShared storage helps after a request. Request coalescing helps while several workers are running at once.The following pseudocode illustrates the pattern:async function readSource(url: string, policy: FreshnessPolicy) { const key = canonicalize(url); const stored = await sourceStore.get(key); if (stored && policy.accepts(stored)) { return stored; } return retrievalLock.run(key, async () => { const latest = await sourceStore.get(key); if (latest && policy.accepts(latest)) { return latest; } const result = await reader.fetch(url, { etag: latest?.etag, lastModified: latest?.lastModified, }); return sourceStore.save(key, result); });}The second cache check is important. Another worker may complete the retrieval while the current worker is waiting for the lock.A real implementation also needs timeouts, lock expiry, access controls, and protection against untrusted page content. The principle remains: parallel reasoning should not automatically create parallel requests for identical material.Canonicalisation also needs care. Removing analytics parameters is usually sensible; deleting every query parameter is not. On many sites, a parameter identifies a genuinely different document or version.Decide Freshness from the Task“Real time” is not a useful universal cache policy.A service-status page may become stale within minutes. A historical standard might remain suitable for years. A product-pricing page can require a shorter refresh window than an archived technical paper.Define freshness at the task or source-class level. The policy can consider the source type, the user’s decision, the content’s known update pattern, and the cost of serving outdated information.When stored content is no longer fresh enough, a conditional request may avoid downloading it again. Servers that support ETag or Last-Modified can report that a resource has not changed. Google's crawl-budget guidance recommends HTTP caching mechanisms for this reason.Where a publisher offers a documented API, feed, repository clone, or bulk export, that route may be more efficient than repeatedly processing presentation-oriented pages. The correct choice still depends on the task and the applicable access conditions.The goal is not the fewest possible requests. It is sufficiently current evidence with as little redundant work as practical.Preserve Evidence Lineage, Including FailureRetrieval efficiency and answer quality are closely linked.Suppose an agent cannot access a maintainer’s explanation of a software vulnerability. It finds a general news article instead and produces a fluent summary. Unless the system preserves the failed retrieval, the reader may never know that the primary explanation was not inspected.Your evidence model should distinguish at least three outcomes:The source was read and supported a claim.The source was read but did not contain relevant evidence.The source could not be read.These states should influence generation and citation. A secondary report may provide useful context, but it should not silently replace unavailable primary evidence.Deduplication requires the same discipline. Five news sites repeating one press release are five URLs but one evidence lineage. An independent analysis that reaches a similar conclusion is a separate piece of evidence.This is why evidenceGroup belongs in the source record. It prevents the model from treating syndication as independent confirmation while keeping every source URL available for attribution.Search First, Then Read SelectivelyDiscovery and extraction are different jobs.Discovery asks which sources are likely to answer the question. Extraction reads the selected pages or documents deeply enough for the model to use them. Keeping these stages separate lets the system narrow its evidence set before requesting full content.Cloudsway Search follows this separation through SmartSearch and Reader. SmartSearch discovers relevant, source-backed information. Reader extracts structured material from selected pages and documents.Those capabilities provide the inputs to a retrieval layer. They do not remove the application’s responsibilities. Your system still needs to coordinate parallel workers, decide freshness, retain source state, and explain when access fails.A search API also does not guarantee that every relevant page is available, reduce total crawling by itself, or establish permission to reuse every result. Those outcomes depend on the provider’s infrastructure, the source’s access rules, and the behaviour of the application.Treat Access Preferences and Permission SeparatelyA responsible retrieval layer needs to understand more than HTTP success.Robots.txt communicates crawler preferences, but it is not an authorisation system. RFC 9309 explicitly separates the protocol from access authorisation. Authentication, rate limits, service terms, and permission for downstream reuse remain distinct concerns.The purpose of a request matters too. Training crawlers, search crawlers, and user-triggered retrieval do different work. OpenAI’s crawler documentation, for example, identifies GPTBot, OAI-SearchBot, and ChatGPT-User separately.Your application may not operate any of those crawlers, but the distinction is useful when defining its own identity and behaviour. A publisher should be able to understand who is requesting content and, where practical, why.Retries also need boundaries. An unavailable source should not trigger an aggressive loop across several workers. Use bounded retries with backoff, preserve the failure, and allow the agent to continue with an explicit evidence gap.Review Retrieval Before Scaling the AgentBefore deploying a research workflow broadly, trace several complete tasks. Look beyond model tokens and response latency.Share retrieval state across every worker in the task.Define freshness by source type and decision risk.Coalesce concurrent reads and use conditional requests where supported.Preserve access failures and group copies by evidence lineage.Measure duplicate fetch rate, cache reuse, primary-source failures, and distinct evidence per retrieval.If a source was refreshed simply because a new agent asked, the policy is coupled to execution rather than information freshness.A Retrieval Layer Is Part of the AnswerWeb-connected AI depends on organisations and individuals continuing to publish useful information. Efficient retrieval helps those sources, but it also protects the quality of the application built on top of them.A shared retrieval layer gives parallel agents a common memory of what has been read. Selective extraction reduces unnecessary page processing. Freshness policies prevent both stale evidence and reflexive refetching. Evidence lineage stops copies from becoming false confirmation. Explicit failure states prevent unavailable sources from disappearing from the reasoning process.None of these mechanisms guarantees complete or permissionless access to the web. Together, they create a more honest and efficient foundation for research agents.As agent workflows move from demonstrations to thousands of concurrent tasks, retrieval decisions stop being implementation details. They become part of the system’s reliability, and part of the answer it ultimately gives.This story is published under the Generative AI publication. Connect with us on LinkedIn and follow Zeniteq to stay in the loop with the latest AI stories. Let’s shape the future of AI together!The Web Was Built to Be Read. AI Is Learning to Consume It. was originally published in Generative AI on Medium, where people are continuing the conversation by highlighting and responding to this story.Source: Generative AI Pub — Published — Category: Image AI