
How AI crawlers scrape the internet at large just changed
For decades, the Internet has operated on a simple exchange: search engines index content and direct users back to original websites, generating traffic and ad revenue for websites of all sizes. This cycle rewards creators that produce quality content with money and a following, while helping users discover new and relevant information. That model is now broken. AI crawlers collect content like text, articles, and images to generate answers, without sending visitors to the original source – depriving content creators of revenue, and the satisfaction of knowing someone is viewing their content. If the incentive to create original, quality content disappears, society ends up losing, and the future of the Internet is at risk.
'If the Internet is going to survive the age of AI, we need to give publishers the control they deserve and build a new economic model that works for everyone – creators, consumers, tomorrow's AI founders, and the future of the web itself,' said Matthew Prince, co-founder and CEO of Cloudflare. 'Original content is what makes the Internet one of the greatest inventions in the last century, and it's essential that creators continue making it. AI crawlers have been scraping content without limits. Our goal is to put the power back in the hands of creators, while still helping AI companies innovate. This is about safeguarding the future of a free and vibrant Internet with a new model that works for everyone.'
'Cloudflare's innovative approach to block AI crawlers is a game-changer for publishers and sets a new standard for how content is respected online. When AI companies can no longer take anything they want for free, it opens the door to sustainable innovation built on permission and partnership,' said Roger Lynch, CEO of Condé Nast. 'This is a critical step toward creating a fair value exchange on the Internet that protects creators, supports quality journalism and holds AI companies accountable.'
'We have long said that AI platforms must fairly compensate publishers and creators to use our content. We can now limit access to our content to those AI partners willing to engage in fair arrangements,' said Neil Vogel, CEO of Dotdash Meredith. 'We're proud to support Cloudflare and look forward to using their tools to protect our content and the open web.'
'As the largest publisher in the country, comprised of USA TODAY and over 200 local publications throughout the USA TODAY Network, blocking unauthorized scraping and the use of our original content without fair compensation is critically important,' said Renn Turiano, Chief Consumer and Product Officer of Gannett Media. 'As our industry faces these challenges, we are optimistic the Cloudflare technology will help combat the theft of valuable IP.'
'Creators and publishers around the world leverage Pinterest to expand their businesses, reach new audiences and directly measure their success. As AI continues to reshape the digital landscape, we are committed to building a healthy Internet infrastructure where content is used for its intended purpose, so creators and publishers can thrive,' said Bill Ready, CEO of Pinterest.
'AI companies, search engines, researchers, and anyone else crawling sites have to be who they say they are. And any platform on the web should have a say in who is taking their content for what,' said Steve Huffman, co-founder and CEO of Reddit. 'The whole ecosystem of creators, platforms, web users and crawlers will be better when crawling is more transparent and controlled, and Cloudflare's efforts are a step in the right direction for everyone.'
'We applaud Cloudflare for advocating for a sustainable digital ecosystem that benefits all stakeholders — the consumers who rely on credible information, the publishers who invest in its creation, and the advertisers who support its dissemination,' said Vivek Shah, CEO of Ziff Davis.
Cloudflare powers one of the world's largest networks, helping to manage and protect traffic for 20% of the web. The company handles trillions of requests daily, and thus has the world's most advanced bot management solutions, accurately distinguishing between human users and AI crawlers. In September 2024, Cloudflare introduced the option to block AI crawlers in a single click. More than one million customers have since chosen this option, meant to be an aggressive but easy solution that halts scraping while they determine their AI strategy.
Now, Cloudflare is taking the next step to enforce a permission-based model for AI crawlers. AI companies will now be required to obtain explicit permission from a website before scraping. Upon sign-up with Cloudflare, every new domain will now be asked if they want to allow AI crawlers, giving customers the choice upfront to explicitly allow or deny AI crawlers access. This significant shift means that every new domain starts with the default of control, and eliminates the need for webpage owners to manually configure their settings to opt out. Customers can easily check their settings and enable crawling at any time if they want their content to be freely accessed.
Now Cloudflare is making the content ecosystem more transparent for AI companies and creators. The company recently proposed new ways for AI bots to authenticate themselves as well as for websites to identify those bots – giving creators and website owners new identification mechanisms and control over what crawlers they want to allow. Cloudflare is participating in the development of a new protocol to provide bot owners and AI agent developers with a public, standard way to identify themselves.
Hashtags

Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles


Gulf Business
27 minutes ago
- Gulf Business
Trump's One Big Beautiful Bill explained
Image: Getty Images In July 2025, the US Congress enacted the sweeping One Big Beautiful Bill (OBBB): a landmark legislative overhaul combining permanent extensions of Trump-era tax cuts for individuals and businesses with major spending cuts to welfare programmes and a hefty increase in defense and border security outlays. The bill narrowly cleared its final hurdle in the House of Representatives, positioning it to become law following his signature on July 4. According to the add approximately $3.3–$ 3.4 trillion to federal deficits over the next decade and leave 11–12 million Americans without Medicaid coverage, a claim strongly disputed by the White House. 'President Trump's One Big, Beautiful Bill delivers on the commonsense agenda that nearly 80 million Americans voted for – the largest middle-class tax cut in history, permanent border security, massive military funding, and restoring fiscal sanity. The pro-growth policies within this historic legislation are going to fuel an economic boom like we've never seen before. President Trump looks forward to signing the One Big, Beautiful Bill into law to officially usher in the Golden Age of America,' the VICTORY: The One Big Beautiful Bill Passes U.S. Congress, Heads to President Trump's Desk 🇺🇸🎉 — The White House (@WhiteHouse) From a B2B perspective, this bill sends strong signals: a brighter corporate tax landscape and investment clarity, contrasted with harsh reductions in healthcare and social safety nets. It deliberately reshapes incentives in sectors like renewable energy, defense, manufacturing, and infrastructure, offering strategic opportunities for businesses and investors. With permanent 2017 tax cuts, expanded bonuses, and full 100% expensing, the bill aims to stimulate corporate investment. Yet it simultaneously reverses many climate-related credits, potentially chilling solar and wind projects . Defense and security sectors, by contrast, are set to benefit from a . Lost in the US-centric coverage, however, are ripple effects in the GCC region, from fiscal and investment flows to energy markets and defense partnerships. Gulf sovereign wealth funds with heavy US bond and equity exposure may see altered yields and investment valuations. A return to robust US fossil fuel production and weaker renewables support could benefit GCC oil exporters, even as geopolitical and military collaboration dynamics evolve. Sector-Wise Breakdown Tax & Corporate Sector Permanent tax cuts : Lowers corporate and individual tax rates, increases Business certainty : Enhanced planning through long-term tax predictability, including 100% Section 179 expensing. Trade & remittance levy : Introduces a 1% tax on remittances—raising potential issues for global fund flows . Healthcare & Welfare Drastic Medicaid/SNAP cuts : Deep reductions could strip about 10–11 million low-income Americans of benefits. Eligibility changes : Programmes now include stricter work mandates and state cost-sharing, potentially straining hospital systems. Defense & Border Security Defense boost : +$150 billion for military, including 'Golden Dome' missile defense, drones, and nuclear upgrades. Immigration enforcement : +$150 billion for border control, ICE expansion, detention capacity for up to 1 million deportees annually. Energy & Environment Clean energy rollback : Repeals IRA tax credits, halts renewables momentum, and favors fossil fuels, nuclear, and gas sectors . Energy dominance push : Positions US around nuclear and gas reliability; delays investment in solar and wind . Infrastructure & Tech ATC modernisation : . Spectrum & R&D incentives : 600 MHz spectrum auction planned by 2034; R&D expensing restored to spur innovation . Agriculture & Rural Support for rural hospitals : $50 billion allocated to support struggling healthcare systems in non-urban areas . Agricultural enhancements : Elevated crop insurance, price supports, and disaster relief totalling approximately $60 billion . GCC Impact Snapshot Sovereign wealth & portfolio returns: The tax cuts and increased US debt may drive higher bond yields, squeezing GCC external debt issuances. A new remittance tax could also slightly dent returns for GCC-based investors in the US. Energy market ripples: Rollbacks in clean energy tilt US fuel demands back to oil and gas, supporting GCC hydrocarbon export prices in the short to mid-term . Defense & security ties: Expanded US defense budgets open avenues for GCC collaboration on advanced military and border technologies. Investment climate: Tax clarity may attract more GCC foreign direct investment into US infrastructure and technology sectors, though uncertainty in welfare and fiscal policy might temper risk appetite. Trump's 'One Big Beautiful Bill' epitomises a high-stakes gamble: it locks in permanent tax relief and certainty for corporations and the wealthy, while significantly slashing social safety nets, primarily Medicaid, potentially leaving nearly 12 million Americans uninsured. Although fossil fuel industries benefit from revived incentives, the rollback of clean‑energy credits casts a shadow over green energy's momentum, even as targeted investments in technology, defense, and research & development open long‑term growth pathways, assuming fiscal discipline and stable global trade persist.


Zawya
35 minutes ago
- Zawya
Gold poised for weekly gain on dollar weakness, safe-haven demand
Gold prices rebounded on Friday and were heading for a weekly gain, helped by a retreat in the U.S. dollar and safe-haven inflows as U.S. President Donald Trump's deadline for trade deals loomed. Spot gold was up 0.3% to $3,334.47 per ounce, as of 1002 GMT. The precious metal is up about 1.9% this week. U.S. gold futures gained 0.1% to $3,344.70. The dollar index slipped 0.2% and was on track for a second week of decline, making gold less expensive for other currency holders. "The apprehension about the fiscal situation in the U.S. (after Trump's sweeping tax-cut bill passed Congress) and the lingering uncertainty over the approaching July 9 deadline for the tariff issue has boosted safe-haven demand," said Ricardo Evangelista, senior analyst at brokerage firm ActivTrades. Trump announced that Washington will start sending letters to countries on Friday, marking a shift from earlier plans for individual trade deals. On April 2, he announced reciprocal tariffs of 10%-50%, but later reduced most to 10% until July 9 to allow for negotiations. Meanwhile, Trump's tax-cut legislation cleared its final hurdle in Congress on Thursday, making his 2017 cuts permanent, funding his immigration crackdown, and adding new tax breaks promised during Trump's 2024 campaign. Data showed U.S. job growth was unexpectedly solid in June, but nearly half of the increase in nonfarm payrolls came from the government sector, with private industry gains the smallest in eight months as businesses battled rising economic headwinds. "The latest U.S. payroll data supports the case of a slowdown of the economy, but no standstill, slowing the pressure on the Fed to cut interest rates anytime soon," said UBS commodity analyst Giovanni Staunovo. Elsewhere, spot silver edged 0.1% lower to $36.78 per ounce and palladium eased 0.3% to $1,133.21. Platinum rose 0.5% to $1,374.21 per ounce and was heading for its fifth straight week of gains. (Reporting by Brijesh Patel in Bengaluru, additional reporting by Ishaan Arora; Editing by Harikrishnan Nair)


The National
2 hours ago
- The National
US imposes new sanctions against Iran oil trade and Lebanon's Hezbollah
The US has imposed new sanctions against a network that smuggles Iranian oil falsely presented as Iraqi oil and on a Hezbollah-controlled financial institution, the Treasury Department said. A network of companies run by Iraqi-British national Salim Ahmed Said has been buying and shipping billions of dollars worth of Iranian oil disguised as, or blended with, Iraqi oil since at least 2020, the department said. 'Treasury will continue to target Tehran 's revenue sources and intensify economic pressure to disrupt the regime's access to the financial resources that fuel its destabilising activities,' Treasury Secretary Scott Bessent said. The US has imposed layers of sanctions on Iran's oil exports over its nuclear programme and funding of militias across the Middle East. 'As President Trump has made clear, Iran's behaviour has left it decimated. While it has had every opportunity to choose peace, its leaders have chosen extremism,' added Mr Bessent. The Treasury Department also issued sanctions against seven senior officials and one entity linked to Al Qard Al Hassan (AQAH), a Hezbollah-controlled financial institution. The armed Hezbollah group is Iran's proxy in Lebanon. 'These officials, through their management roles, have facilitated Hezbollah's evasion of sanctions, enabling AQAH to conduct millions of dollars in transactions through 'shadow' accounts,' said State Department spokeswoman Tammy Bruce. This comes as the Trump administration intensifies efforts to pressure Tehran into returning to nuclear negotiations and to clamp down on its proxies across the region. The US carried out strikes on June 22 on three Iranian nuclear sites including its most deeply buried enrichment plant, Fordow. The Pentagon said on Wednesday the strikes had degraded Iran's nuclear programme by up to two years, despite a far more cautious initial assessment that had leaked to the public. The US and Iran were expected to hold talks about its nuclear programme next week in Oslo, Axios reported. Said's companies and vessels blend Iranian oil with Iraqi oil, which is then sold to western buyers through Iraq or the Gulf as purely Iraqi oil using forged documentation to avoid sanctions, Treasury said. Said controls Gulf-based company VS Tankers, although he avoids formal association with it, the Treasury said. Formerly known as Al Iraqia Shipping Services & Oil Trading (Aissot), VS Tankers has smuggled oil for the benefit of the Iranian government and the Islamic Revolutionary Guard Corps, which is designated by Washington as a terrorist organisation, it said. The sanctions block US assets of those designated and prevent Americans from doing business with them. Hezbollah trying to rebuild its operations The Lebanese armed group Hezbollah has been severely weakened following a year-long war with Israel that ended in November. The US Treasury said the new sanctions underscore a commitment to disrupting the group's sanctions evasion schemes and supporting efforts by the new Lebanese government to limit Hezbollah's influence. 'Through their roles at AQAH, these officials sought to obfuscate Hezbollah's interest in seemingly legitimate transactions at Lebanese financial institutions, exposing these banks to significant AML/CFT [money laundering and terrorism financing] risk while allowing Hezbollah to funnel money for its own benefit,' said Deputy Secretary Michael Faulkender. 'As Hezbollah seeks money to rebuild its operations, Treasury remains strongly committed to dismantling the group's financial infrastructure and limiting its ability to reconstitute itself,' he added. Hezbollah has long been described as a 'state within a state' in Lebanon, maintaining its own military, security and social infrastructure that operates independently of government control. It also runs a parallel financial system, receiving funding from Iran and various independent sources, while managing its finances outside official state institutions. Ms Bruce said that Washington remains committed to supporting Lebanon by disrupting schemes that empower Hezbollah's destabilising influence. 'We will continue to employ all available tools to ensure that this terrorist group no longer poses a threat to the Lebanese people and the region,' she said.