The battle to AI-enable the web: NLweb and what enterprises need to know

Business Mayor

26-05-2025

Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More
In the first generation of the web, back in the late 1990s, search was okay but not great, and it wasn't easy to find things. That led to the rise of syndication protocols in the early 2000s, with Atom and RSS (Really Simple Syndication) providing a simplified way for website owners to make headlines and other content easily available and searchable.
In the modern era of AI, a new group of protocols is emerging to serve the same basic purpose. This time, instead of making sites easier for humans to find, it's all about making websites easier for AI. Anthropic's Model Control Protocol (MCP), Google's Agent2Agent and large language models/ LLMs.txt are among the existing efforts.
The newest protocol is Microsoft's open-source NLWeb (natural language web) effort, which was announced during the Build 2025 conference. NLWeb is also directly linked to the first generation of web syndication standards, as it was conceived and created by RV Guha, who helped create RSS, RDF (Resource Description Framework) and schema.org.
NLWeb enables websites to easily add AI-powered conversational interfaces, effectively turning any website into an AI app where users can query content using natural language. NLWeb isn't necessarily about competing with other protocols; rather, it builds on top of them. The new protocol uses existing structured data formats like RSS, and each NLWeb instance functions as an MCP server.
'The idea behind NLWeb is it is a way for anyone who has a website or an API already to very easily make their website or their API an agentic application,' Microsoft CTO Kevin Scott said during his Build 2025 keynote. 'You really can think about it a little bit like HTML for the agentic web.'
NLWeb transforms websites into AI-powered experiences through a straightforward process that builds on existing web infrastructure while leveraging modern AI technologies.
Building on existing data: The system begins by leveraging structured data that websites already publish, including markup, RSS feeds and other semi-structured formats that are commonly embedded in web pages. This means publishers don't need to rebuild their content infrastructure completely.
Data processing and storage: NLWeb includes tools for adding this structured data to vector databases, which enable efficient semantic search and retrieval. The system supports all major vector database options, allowing developers to choose the solution that best fits their technical requirements and scale.
Read More Building a 'virtual Vegas' in honor of CES
AI enhancement layer: LLMs then enhance this stored data with external knowledge and context. For instance, when a user queries about restaurants, the system automatically layers on geographic insights, reviews and related information by combining the vectorized content with LLM capabilities to provide comprehensive, intelligent responses rather than simple data retrieval.
Universal interface creation: The result is a natural language interface that serves both human users and AI agents. Visitors can ask questions in plain English and receive conversational responses, while AI systems can programmatically access and query the site's information through the MCP framework.
This approach allows any website to participate in the emerging agentic web without requiring extensive technical overhauls. It makes AI-powered search and interaction as accessible as creating a basic webpage was in the early days of the internet.
There are a lot of different protocols emerging in the AI space; not all do the same thing.
Google's Agent2Agent , for example, is all about enabling agents to talk to each other. It's about orchestrating and communicating agentic AI and is not particularly focused on AI-enabling existing websites or AI content. Maria Gorskikh, founder and CEO of AIA and a contributor to the Project NANDA team at MIT, explained to VentureBeat that Google's A2A enables structured task passing between agents using defined schemas and lifecycle models.
'While the protocol is open-source and model-agnostic by design, its current implementations and tooling are closely tied to Google's Gemini stack — making it more of a backend orchestration framework than a general-purpose interface for web-based services,' she said.
Another emerging effort is LLMs.txt. Its goal is to help LLMs better access web content. While on the surface, it might sound somewhat like NLWeb, it's not the same thing.
'NLWeb doesn't compete with LLMs.txt; it is more comparable to web scraping tools that try to deduce intent from a website,' Michael Ni, VP and Principal Analyst at Constellation Research told VentureBeat.
Krish Arvapally, co-founder and CTO of Dappier, explained to VentureBeat that LLMs.txt provides a markdown-style format with training permissions that helps LLM crawlers ingest content appropriately. NLWeb focuses on enabling real-time interactions directly on a publisher's website. Dap pier has its own platform that automatically ingests RSS feeds and other structured data, then delivers branded, embeddable conversational interfaces. Publishers can syndicate their content to their data marketplace.
MCP is the other big protocol, and it is increasingly becoming a de facto standard and a foundational element of NLWeb. Fundamentally, MCP is an open standard for connecting AI systems with data sources. Ni explained that in Microsoft's view, MCP is the transport layer, where, together, MCP and NLWeb provide the HTML and TCP/IP of the open agentic web.
Forrester Senior Analyst Will McKeon-White sees a number of advantages for NLWeb over other options.
'The main advantage of NLWeb is better control over how AI systems 'see' the pieces that make up websites, allowing for better navigation and more complete understanding of the tooling,' McKeon-White told VentureBeat. 'This could reduce both errors from systems misunderstanding what they're seeing on websites, as well as reduce interface rework.'
Microsoft didn't just throw NLWeb over the proverbial wall and hope someone would use it.
Microsoft already has multiple organizations engaged and using NLWeb, including Chicago Public Media, Allrecipes, Eventbrite, Hearst (Delish), O'Reilly Media, Tripadvisor and Shopify.
Andrew Odewahn, Chief Technology Officer at O'Reilly Media is among the early adopters and sees real promise for NLWeb.
'NLWeb leverages the best practices and standards developed over the past decade on the open web and makes them available to LLMs,' Odewahn told VentureBeat. 'Companies have long spent time optimizing this kind of metadata for SEO and other marketing purposes, but now they can take advantage of this wealth of data to make their own internal AI smarter and more capable with NLWeb.'
In his view, NLWeb is valuable for enterprises both as consumers of public information and publishers of private information. He noted that nearly every company has sales and marketing efforts where they might need to ask, 'What does this company do?' or 'What is this product about?'
'NLWeb provides a great way to open this information to your internal LLMs so that you don't have to go hunting and pecking to find it,' Odewahn said. 'As a publisher, you can add your own metadata using schema.org standard and use NLWeb internally as an MCP server to make it available for internal use.'
Using NLWeb isn't necessarily a heavy lift, either. Odewahn noted that many organizations are probably already using many of the standards NLWeb relies on.
'There's no downside in trying it out now since NLWeb can run entirely within your infrastructure,' he said. 'It's open source software meeting the best in open source data, so you have nothing to lose and a lot to gain from trying it now.'
Constellation Research Analyst Michael Ni has a somewhat positive viewpoint on NLWeb. However, that doesn't mean enterprises need to adopt it immediately.
Ni noted that NLWeb is in the very early stages of maturity and enterprises should expect 2-3 years for any substantial adoption. He suggests that leading-edge companies with specific needs, such as active marketplaces, can look to pilot with the ability to engage and help shape the standard.
'It's a visionary specification with clear potential, but it needs ecosystem validation, implementation tooling, and reference integrations before it can reach mainstream enterprise pilots,' Ni said.
Others have a somewhat more aggressive viewpoint on adoption. Gorskikh suggests taking an accelerated approach to ensure your enterprise doesn't fall behind.
'If you're an enterprise with a large content surface, internal knowledge base, or structured data, piloting NLWeb now is a smart and necessary step to stay ahead,' she said. 'This isn't a wait-and-see moment — it's more like the early adoption of APIs or mobile apps.'
That said, she noted that regulated industries need to tread carefully. Sectors like insurance, banking and healthcare should hold off on production use until there's a neutral, decentralized verification and discovery system in place. There are already early-stage efforts addressing this — such as the NANDA project at MIT that Gorskikh participates in, which is building an open, decentralized registry and reputation system for agentic services.
For enterprise AI leaders, NLWeb is a watershed moment and a technology that should not be ignored.
AI is going to interact with your site, and you need to AI enable it. NLWeb is one way that will be particularly attractive to publishers, much like RSS became a must-have for all websites in the early 2000s. In a few years, users will just expect it to be there; they will expect to be able to search and find things, while agentic AI systems will need to be able to access the content as well.
That's the promise of NLWeb.