Cloudflare blocks AI crawlers by default in a major win for web publishers
Historical context: How it used to work
Traditionally, the web operated on a 'content-for-traffic' model: search engines crawled sites and sent visitors back via links. This model supported ad revenue and visibility. With the rise of generative AI, firms have harvested vast amounts of web content like text, images, code, without driving any traffic back to publishers.
What Cloudflare is doing now
New Cloudflare users are now opted‑out of AI scraper access by default. Owners must explicitly choose to allow bots. An optional Pay Per Crawl program is also in development: publishers can monetize access by charging AI companies per crawl. Major players like Time, Reddit, BuzzFeed, and The Atlantic have already expressed support.
How this affects AI companies
AI firms relying on large language models such as OpenAI, Google, Meta, and Anthropic will face new hurdles. They must now seek permission or negotiate access fees, reducing free access to diverse data. The rise of 'AI Labyrinth,' a honeypot of decoy pages, further complicates unregulated scraping.
Technical and operational roadblocks
AI scrapers pose challenges. They ignore existing protections like robots.txt, cause server strain, and increase bandwidth costs. Cloudflare's answer includes advanced bot detection powered by behavioural analysis and ML, plus the labyrinth tactic to confuse scrapers.
Industry reactions and implications
Content owners, including Universal Music Group and Reddit, have welcomed the change, calling it essential to sustainable web economics. Cloudflare's CEO Matthew Prince summed it up: 'If the Internet is going to survive the age of AI, we need to give publishers the control they deserve'. But critics worry that a permission-based web could limit AI innovation.
New web marketplace on the horizon
Cloudflare is exploring a content-access marketplace where AI firms pay for usage, transforming free scrapes into transactions. This could value content by data rather than clicks, offering publishers a fresh revenue channel.
Cloudflare's move shifts control back to creators, forcing a rethink of how AI companies gather data. Publishers now have the power to block, charge, or allow bots. This shift may define the economic and ethical foundations of the AI-powered web.
Hashtags
Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
India.com
an hour ago
- India.com
Rs 86000000 in salary: Google, Meta, and OpenAI ready to offer huge money for people with talent in...
Rs 86000000 in salary: Google, Meta, and OpenAI ready to offer huge money for people with talent in… Top Tier Talent Salary: In order to hire special talents, major companies around the world are making changes in their salary structure. Tech giant Google has also made major changes in the way the company pays salaries. It has made changes in the salary to attract good and talented employees. These moves are necessary to stay ahead in the ongoing competition in the field of Artificial Intelligence. Not only Google and Meta but also OpenAI are also offering huge salary packages to talented employees. As per a report by Business Insider, citing US Department of Labour documents, software engineers at Google can get a basic salary of USD34,0000 (approx Rs 3 crore). Apart from the basic salary, the company will also give its shares and bonuses, further increasing the total income. Notably, positions such as Product managers, AI researchers and people working in other technical positions are getting impressive salary packages. Meta Is Offering Huge Salary Google is currently facing a very tough competition from other tech giants such as Meta and OpenAI. These companies are also luring good talents in AI by offering them huge salaries. Meta has also invested huge amount in AI and now hiring AI researchers and engineers to power its Generative AI and Reality Labs division. Meta's significant investment in advanced AI in 2023 is reflected in high salaries for its senior AI researchers, ranging from USD600,000 to USD1 million per anum. The salary also includes bonuses and stock options. OpenAI, backed by Microsoft, also offers competitive packages for senior research engineers. These packages range from USD200,000 to USD370,000 in base salary, reaching USD800,000 to USD1 million with equity and profit-sharing incentives. Why Is There A Salary Increase? The tech giants are increasing the salary packages because they want to keep employees who are capable of enhancing large language models, improving generative AI tools, and developing new technologies. According to experts, these high salary packages are not just for new hires but are also to retain good employees in the company.
Time of India
2 hours ago
- Time of India
Leo Daily Horoscope Today (Aaj Ka Singh Rashifal), July 13, 2025: Rest now to see more clearly
Today calls for calmness and rest. Instead of pushing harder, slow down and breathe. Taking a break now will help sharpen your vision tomorrow. Do not worry about missing out. The universe is still working behind the scenes. Rest is not a waste of time, it is preparation. When you pause today, you will move with clarity and purpose in the days to come. Leo Love Horoscope Today In love, your heart may need some quiet. If recent days were filled with misunderstandings or emotional confusion, today is the moment to simply be still with your feelings. Give space to your partner and to yourself. Communication can wait till the mood becomes peaceful. If single, do not rush into connections. Use this time to reflect on what kind of love truly serves your soul. The love that is meant for you will wait, and it will match your peace. Leo Career Horoscope Today At work, slow down your pace even if the pressure feels high. There is no prize for burning out. Instead, take stock of what's important and pause before committing to anything big. You may discover that a quiet break gives you better solutions than overthinking. Focus on finishing one task well rather than doing many half-heartedly. by Taboola by Taboola Sponsored Links Sponsored Links Promoted Links Promoted Links You May Like Is it legal? How to get Internet without paying a subscription? Techno Mag Learn More Undo Sometimes, the best decision is to step back and observe. Your clarity will return, and your creative mind will benefit from this calm. Leo Money Horoscope Today Financially, it is a good day to review and reflect instead of acting quickly. Do not make large investments or take risks out of urgency. Look over your accounts slowly and see where you can simplify. Small savings or reducing a needless expense will be more beneficial today than hunting for gains. This is a day to pause, not to panic. Financial clarity will return once your mental space becomes more grounded. Avoid borrowing or lending for now. Leo Health Horoscope Today Your health may feel a little tired or off-balance today. Instead of ignoring signs like body stiffness, headaches, or restlessness, respond with care. Your body is asking you for recovery. Take a short nap, eat light food, and avoid loud places. Warm water and a peaceful walk in fresh air will help clear your energy. Today is not for pushing limits. It is for healing and protecting your strength. When your body is rested, your inner light shines brighter. Discover everything about astrology at the Times of India , including daily horoscopes for Aries , Taurus , Gemini , Cancer , Leo , Virgo , Libra , Scorpio , Sagittarius , Capricorn , Aquarius , and Pisces .
The Hindu
2 hours ago
- The Hindu
Troubling ascent
In the world of AI chatbots, dominated by OpenAI's ChatGPT, xAI's Grok has rapidly carved out a name for itself, albeit one now mired in a controversy. Designed as an alternative to its 'woke' rivals, Grok's digital tirade, which included praise for Adolf Hitler and controversial comments about Jewish people, ignited a firestorm of criticism. This culminated in the unexpected resignation of X's CEO Linda Yaccarino. While Ms. Yaccarino's public statement provided no specific reason for her exit, it highlighted a crucial, and often confusing, aspect of Elon Musk's corporate structure at X Corp. Grok is a product of xAI, an AI company founded by Mr. Musk in March 2023. xAI and X Corp. (formerly Twitter Inc.) were once sister companies. However, in March, Mr. Musk announced that xAI had acquired X Corp. in an all-stock deal, merging the two into a single entity called Holdings. This merger formalised an already deeply intertwined relationship, effectively putting the position of the CEO of X Corp. at risk. Grok is integrated into the X platform, where it is available to subscribers. It utilises X's real-time data to inform its responses. The disturbing content generated by Grok in early July was a response to a user query about dealing with 'anti-white hate'. The chatbot suggested that Hitler would 'spot the pattern and handle it decisively, every damn time.' In another now-deleted post, Grok echoed anti-Semitic tropes, including the conspiracy theory that Jews control Hollywood, and made offensive remarks about individuals with Jewish-sounding surnames. Just a day after Grok's pro-Hitler comments went viral, Ms. Yaccarino announced her decision to step down as CEO of X. Tightrope walk Ms. Yaccarino, a seasoned advertising executive, was appointed in May 2023 to restore advertiser confidence and bring stability to X. Her two-year tenure was marked by a constant struggle to balance the platform's commitment to 'free speech' with the practical challenges of content moderation and brand safety. Analysts suggested that the 'lack of fit' between her corporate style and Mr. Musk's disruptive approach may have reached a tipping point with the Grok controversy. Grok's behaviour was not an isolated incident. In May, the chatbot became fixated on the racist conspiracy theory of 'white genocide' in South Africa, often responding to unrelated questions about topics such as baseball or HBO. Mr. Musk launched Grok in November 2023, naming it after a term from Robert Heinlein's sci-fi novel Stranger in a Strange Land, which signifies a profound, intuitive understanding. The chatbot was pitched as a 'maximum truth-seeking' alternative, characterised by a 'rebellious streak' that wouldn't shy away from sensitive topics. xAI asserts that Grok 3 outperforms competitors such as OpenAI's GPT-4o in benchmarks related to mathematical and scientific reasoning. Additionally, the model introduces a 'Think' mode, enabling more comprehensive information processing before generating a response. Grok's distinguishing feature lies in its real-time access to the extensive and chaotic information stream of X itself. However, this design choice presents both its unique selling point and its Achilles' heel. An AI learning from a platform where content moderation has been significantly relaxed is akin to unleashing a digital storm. Following the deletion of Hitler-praising posts, Grok's own X account issued an apology, labelling the comments 'an unacceptable error from an earlier model iteration' and condemning Nazism. xAI further clarified that the company is solely training truth-seeking models and utilises feedback from millions of users to identify and enhance the model. In her farewell post, Ms. Yaccarino expressed gratitude to Mr. Musk for the opportunity to work on transforming X Corp. Her departure leaves a significant void and raises new questions about the platform's direction. Grok's intended triumph as an unfiltered AI has instead become a stark illustration of the potential dangers of such an approach. It serves as a sobering reminder that AI's quality is contingent upon the data it learns. In the unregulated expanse of the Internet, a rebellious streak can rapidly escalate into something far more sinister.
