Web giant Cloudflare to block AI bots from scraping content by default
Starting Tuesday, every new web domain that signs up to Cloudflare will be asked if they want to allow AI crawlers, effectively giving them the ability to prevent bots from scraping data from their websites.
Cloudflare is what's called a content delivery network, or CDN. It helps businesses deliver online content and applications faster by caching the data closer to end-users. They play a significant role in making sure people can access web content seamlessly every day.
Roughly 16% of global internet traffic goes directly through Cloudflare's CDN, the firm estimated in a 2023 report.
"AI crawlers have been scraping content without limits. Our goal is to put the power back in the hands of creators, while still helping AI companies innovate," said Matthew Prince, co-founder and CEO of Cloudflare, in a statement Tuesday.
"This is about safeguarding the future of a free and vibrant Internet with a new model that works for everyone," he added.
AI crawlers are automated bots designed to extract large quantities of data from websites, databases and other sources of information to train large language models from the likes of OpenAI and Google.
Whereas the internet previously rewarded creators by directing users to original websites, according to Cloudflare, today AI crawlers are breaking that model by collecting text, articles and images to generate responses to queries in a way that users don't need to visit the original source.
This, the company adds, is depriving publishers of vital traffic and, in turn, revenue from online advertising.Tuesday's move builds on a tool Cloudflare launched in September last year that gave publishers the ability to block AI crawlers with a single click. Now, the company is going a step further by making this the default for all websites it provides services for.
OpenAI says it declined to participate when Cloudflare previewed its plan to block AI crawlers by default on the grounds that the content delivery network is adding a middleman to the system.
The Microsoft-backed AI lab stressed its role as a pioneer of using robots.txt, a set of code that prevents automated scraping of web data, and said its crawlers respect publisher preferences.
"AI crawlers are typically seen as more invasive and selective when it comes to the data they consumer. They have been accused of overwhelming websites and significantly impacting user experience," Matthew Holman, a partner at U.K. law firm Cripps, told CNBC.
"If effective, the development would hinder AI chatbots' ability to harvest data for training and search purposes," he added. "This is likely to lead to a short term impact on AI model training and could, over the long term, affect the viability of models."
Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
Business Insider
31 minutes ago
- Business Insider
OpenAI employees share their 3 favorite tips for using ChatGPT
If you ever happen to see Nick Turley, the head of ChatGPT at OpenAI, muttering to himself on a weekday morning, it might be because he's talking to a chatbot. Turley said that ChatGPT's voice feature is his favorite tip for using the technology on a recent episode of the OpenAI podcast. "On my way to work, I'll use it to process my own thoughts. With some luck, and I think this works most days, I'll have the restructured list of to-dos by the time I actually get there," he said, adding that the voice feature isn't yet mainstream because there are a bunch of small "kinks" still. He said he finds it valuable to force himself to articulate his thoughts aloud, and wants to see the feature improve next year. Mark Chen, OpenAI's chief research officer, said on the podcast that he's a fan of the deep research feature, especially before an introduction. "When I go meet someone new, when I'm going to talk to someone about AI, I just preflight topics," Chen said. "I think the model can do a really good job of contextualizing who I am, who I'm about to meet, and what things we might find interesting." And podcast host Andrew Mayne, who was formerly OpenAI's science communicator and worked on ChatGPT, said he uses the technology when he's out at a restaurant. "I take a photograph of a menu and I'm like, 'Help me plan a meal or whatever, I'm trying to stick to a diet," Mayne said. Turley, however, cautioned against using the same trick for the wine list. "It keeps embarrassing me with hallucinated wine recommendations, and I go order it and they're like, 'Never heard of this one,'" he said. Corporate executives across companies are using AI in their daily lives, and OpenAI CEO Sam Altman is no different. Altman said on the "ReThinking" podcast in January that he uses it in "the boring ways," for things like processing emails and summarizing documents. When Altman spoke on the OpenAI podcast in June, he said that he uses ChatGPT "constantly" as a father. At the time, he said he was mainly using it to research developmental stages. "Clearly, people have been able to take care of babies without ChatGPT for a long time," Altman said. "I don't know how I would have done that."
Yahoo
34 minutes ago
- Yahoo
House GOP leaders scrambling to rally holdouts behind Trump megabill
GOP leaders are scrambling to unite their conference behind the massive Senate bill that would enact President Trump's domestic agenda and get it to the president's desk before the holiday weekend. They have their work cut out for them. Not only are moderate Republicans balking at cuts to Medicaid, which were made larger by the Senate, but conservatives are also up in arms over the massive increase in deficit spending, also increased by the upper chamber. The combination is raising questions about whether GOP leaders can rally enough support to pass the bill, given a slim House majority that allows for only a small handful of defections. 'If you look at the totality of this, I don't believe this delivers what the president, what the administration, were working to deliver on,' said Rep. Chip Roy (R-Texas), a vocal member of the conservative House Freedom Caucus, which is fighting to rein in deficit spending. 'I know why they're going to lobby for it, I know why the president's going to push for it. They want to see it get done, and I get it,' he continued. 'But I think we have more work to do.' The resistance poses a huge challenge for Speaker Mike Johnson (R-La.) and other GOP leaders, who are racing to bring the holdouts on board for the sake of enacting the core of Trump's second-term agenda, including sweeping tax cuts, a crackdown on immigration, a shift away from green energy and huge cuts in federal health and nutrition programs. Some of those House GOP critics — moderates and conservatives alike — had warned in no uncertain terms that they were ready to oppose the bill if it were made 'worse' by the Senate. Now they face the dilemma of either abandoning that position, to give Trump a win, or standing by it to vote 'no' — even if it means sinking the bill. Centrists and hard-liners are now weighing those two forces, and some were quick to acknowledge the pickle they're in. 'Maybe I'll get lucky and have a rough enough landing or something that I'm unable to make [it] to D.C. for a few weeks,' one moderate House Republican told The Hill, underscoring the corner they find themselves in. Hanging over their decision is the president himself, who is pressing hard on Republicans of all stripes to back the legislation — and might encourage primary challenges against recalcitrant lawmakers who oppose the 'big, beautiful bill.' That scenario is not just a hypothetical. Rep. Thomas Massie (R-Ky.) — who voted against the House version of the legislation in May and is expected to oppose the Senate bill this week — has been the target of a MAGA-backed super PAC fighting to get someone more loyal to Trump into his seat. Across the Capitol, Sen. Thom Tillis (R-N.C.) also faced the president's wrath after announcing he would oppose the package. A day later, Tillis announced that he will retire at the end of this term. Democrats aren't making it any easier. While they're powerless to block the legislation, they're highlighting the most controversial provisions of the package, which include steep cuts to low-income health and nutrition programs — money Republicans are using to help pay for their tax cuts. Democrats are already honing their attacks on battleground Republicans who might support the bill. 'Republicans claim to be the party of working-class Americans — you are not. You are hurting working-class Americans. This bill won't make life more affordable for the American people. It will make life more expensive,' House Minority Leader Hakeem Jeffries (D-N.Y.) told reporters Tuesday in the Capitol. 'Why would any so-called traditional Republican, particularly those who are in swing seats all across America, vote for a bill that will aggressively and actively hurt their constituents?' Adding to the pressure, Johnson and GOP leadership are trying to pass the bill by their self-imposed July 4 deadline, which is Friday, a timeline that will require a herculean effort to meet. In a statement right after the Senate vote, the top four House GOP leaders said they would finish their work 'in time for Independence Day.' But later in the day, when talking to reporters in the hallway, Johnson was less sure. 'We'll see what happens in the next 24 hours,' he said. While coy on the timeline, he was clear that he was not thrilled with the Senate's changes — but he suggested the House would not make changes of their own, which would require another vote in the Senate. 'I'm not happy with what the Senate did to our product, but we understand this is the process, it goes back and forth, and we'll be working to get all of our members to 'yes,'' the Speaker said. 'So high stakes, aggressive schedule, and we knew it would come to this moment.' Some lawmakers are skeptical they can get it done in time. 'The changes the Senate made to the House passed Beautiful Bill, including unacceptable increases to the national debt and the deficit, are going to make passage in the House difficult,' Rep. Marlin Stutzman (R-Ind.), a Freedom Caucus member, wrote on the social platform X. 'I am willing to work through the 4th to make sure this bill is better for America's children tomorrow by spending less today.' The sprint, nonetheless, is already underway. The House Rules Committee convened for a meeting about the bill Tuesday afternoon, marking the first step in the consideration process in the lower chamber. Once the panel adopts a procedural rule, House GOP leaders want to convene the chamber at 9 a.m. Wednesday to begin debating the rule, then hold a vote to adopt the procedural rule, followed by a vote on the underlying legislation. But the first step is already on thin ice. Rep. Andy Harris (R-Md.), the chair of the Freedom Caucus, said Tuesday on Fox News that he will vote against the procedural rule on Wednesday, warning that it will be stopped in its tracks — a reality that would bring legislative business to a halt. Republicans can only lose three of their own and still advance the rule, assuming full attendance and united Democratic opposition. Two of those defectors are already claimed with Harris and Rep. Ralph Norman (R-S.C.). 'That's exactly why a group of us are not going to vote to advance the bill until we iron out some of the deficit problems with the bills,' Harris said when asked about Elon Musk's criticism of the bill. 'Look, Mr. Musk is right, we cannot sustain these deficits; he understands finances, he understands debts and deficits, and we have to make further progress. And I believe the Freedom Caucus will take the lead in making that further progress.' Trump weighed in on Tuesday afternoon, praising the Senate for passing his agenda and urging House Republicans to follow suit. 'I thought the Senate was going to be tougher than the House. We got there. We got pretty much what we wanted,' Trump said. 'It's the biggest bill ever passed if we get this done.' A senior White House official was even more direct, all but demanding that the House pass the bill in its current form in time to grant Trump a symbolic signing on Independence Day. The official dismissed the idea that the House could make changes to the Senate bill and then have the chambers 'conference' the two versions into one. 'The end of the road is here. The bill is finished. The bill needs to be sent to the president's desk and it needs to be done … on or before July 4,' the official said. 'I don't think we view conference as a productive enterprise.' The White House is whipping hard ahead of the vote, the official added, with a number of top officials working the phones, including Trump; his budget director, Russell Vought; and the heads of the Energy and Treasury departments. Alex Gangitano contributed reporting. Copyright 2025 Nexstar Media, Inc. All rights reserved. This material may not be published, broadcast, rewritten, or redistributed.
New York Post
34 minutes ago
- New York Post
Top White House official urges probe of Fed's Jerome Powell over deceptive' testimony on 'Palace of Versailles' $2.5B HQ revamp
A top White House official on Wednesday claimed Federal Reserve Chair Jerome Powell could lose his job over his allegedly 'deceptive' testimony to Congress over the central bank's lavish $2.5 billion revamp of its DC headquarters. Federal Housing Finance Agency Director William Pulte, who is also chairman of the two US-backed mortgage lenders Fannie Mae and Freddie Mac, demanded that Powell be probed by lawmakers and suggested he could even be fired by President Trump. 4 FHDA director Bill Pulte, left, has called on Congress to investigate Jerome Powell, right, for his 'deceptive' testimony to the Senate Banking Committee last week over the 'Palace of Versailles' $2.5B revamp. Advertisement 'I am asking Congress to investigate Chairman Jerome Powell, his political bias, and his deceptive Senate testimony, which is enough to be removed 'for cause', Pulte said in a leaked statement obtained by The Post. 'Jerome Powell's $2.5B Building Renovation Scandal stinks to high heaven, and he lied when asked about the specifics before Congress. This is nothing short of malfeasance,' the 37-year-old former journalist and private equity titan added. Powell has already been heavily criticized by President Trump this week, who took the unusual step of sending him a handwritten note urging him to cut interest rates. Advertisement Pulte's call for the Congressional probe comes one week after the Fed chair's appearance before the Senate Banking Committee in which he denied The Post's exclusive reporting about the renovations that have been likened by Sen. Tim Scott (R-SC) to the 'Palace of Versailles.' 'There's no VIP dining room, there's no new marble. There are no special elevators,' Powell insisted under questioning from the powerful panel on Wednesday. 'There are no new water features, there's no beehives, and there's no roof terrace gardens,' Powell said during the grilling by lawmakers. 4 The Post's artwork from its story on Monday which broke the news about how lawmakers were concerend that he may have lied to Congress. Jack Forbes / Donald Pearsall / NY Post Design The 72-year-old's testimony sparked outrage because they directly contradicted the Fed's own planning documents that were signed off by US government pen pushers in 2021 and have not been revised since. Advertisement 'The private dining rooms on Level 4 (of the Fed's Eccles building) will be restored,' reads one excerpt from the filing with the National Capital Planning Commission. 'The Governors' private elevator will be extended to discharge at the dining suite level.' The documents also expressly mention 'vegetated roof terraces' that will welcome 'urban wildlife and pollinators' as well as new marble and water features. A Post editorial published on Tuesday hit out at Powell over 'his privileged arrogance' in the way he spoke to the committee. 4 The Fed's own filings with the National Capital Planning Commission tell a very different story from the one recounted by Jerome Powell to the Senate Banking Committee. NCPC Advertisement He also appeared to dismiss concerns that the revamp was being subsidized by American taxpayers in Wednesday's hearing, saying simply that 'the cost overruns are what they are.' The eye-watering price tag for the overhaul has already ballooned by 30% from an original estimate of $1.9 billion. After The Post broke the story in April about the Fed's reckless spending on the HQ upgrade, former Department of Government Efficiency chief Elon Musk called the news 'an eyebrow raiser.' The Tesla titan, who has since left government, said DOGE should 'definitely' investigate how so much money came to be blown on the glorified vanity project. By comparison, JPMorgan's new headquarters in Midtown Manhattan — a luxe, 60-story tower at 270 Park Ave. designed by star architect Norman Foster — is set to cost an estimated $3 billion. 4 Sen. Tim Scott (R-SC) said the renovations with their lavish amenities look like they belong in the 'Palace of Versailles', the official residence of the long deposed French monarchy. AP The revelations are controversial at a time when the Fed is struggling with mounting losses, which stand at a total of $233 billion from the past three years. Its interest costs surged and outstripped its earnings on bonds it owns when Powell hiked rates in trying to tame rampant inflation during the Biden administration. Advertisement It sank into the red for the first time in its history, posting losses of $114.6 billion in 2023. Officials there insist that losing money in no way impacts their ability to operate and conduct monetary policy. When the Fed makes a profit, that money is passed on to the US Treasury to become part of the federal government's budget. The losses are bundled together in what is known as the Fed's 'deferred asset' that it must pay down before money can be spent on other things, such as defense, education and Medicare.
