AI audits and 'pay per crawl': How Cloudflare is trying to fix a 'broken' web model
Cloudflare said publishers using the cloud company's tools for hosting websites will now block AI crawlers by default from accessing and poaching content without permission.
'Upon sign-up with Cloudflare, every new domain will now be asked if they want to allow AI crawlers, giving customers the choice upfront to explicitly allow or deny AI crawlers access,' the company said.
'This significant shift means that every new domain starts with the default of control, and eliminates the need for webpage owners to manually configure their settings to opt out.'
Cloudflare first addressed concerns about AI data-scraping last year when it gave websites the option to block AI companies from poaching content.
'Now by default you have control over who crawls your site and what that information is used for,' said Stephanie Cohen, a chief strategy officer for Cloudflare.
'The benefit of that is that it creates the conditions for a new business model of the internet to develop,' she told The National after Cloudflare introduced the new settings and options.
Some of the strongest proponents of AI tools, and many of the tools' creators, have justified data scraping, saying it is akin to the early days of search engines when controversy briefly surfaced over whether or not search companies should be able to index sites.
Others say that comparison isn't appropriate, because search engines didn't poach the contents of entire websites.
Additionally, during the early days of web browsers, search engines and the crawlers they implemented provided a framework that built much of the internet as we know it.
It was a win-win situation for the likes of Google and media companies which provided information and sought to attract audiences by delivering web traffic through internet searches.
The debut of OpenAI's ChatGPT in 2022 and other AI platforms turned that economic model on its head.
Instead of directing traffic to websites, AI summaries have quickly become a destination unto themselves, siphoning traffic from the same websites from which they scrape data.
Ms Cohen said publishers and content creators using Cloudflare's services soon noticed a major dip in web traffic.
'Not only was it getting more difficult to get web traffic – to the tune of it being 10 times harder – but it was also getting more difficult at a faster and faster rate,' she said.
The web's economics based on search that built up over the last decade, she said, started to erode over a period of six months.
In 2024, Ms Cohen said Cloudflare allowed users to see which AI companies were scraping their sites and turn off that ability. This year they are taking things further by introducing 'pay per crawl'.
The tool gives publishers and website operators the option of allowing AI scraping for free, charging for it 'at the configured domain-wide price,' or blocking scraping entirely.
As AI developments quickens, so too does the bad blood between media organisations and the tech firms driving the AI boom.
Several lawsuits have been filed. The New York Times has sued OpenAI and Microsoft for allegedly using its articles to power increasingly popular chatbots.
Hashtags
Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
Khaleej Times
12 minutes ago
- Khaleej Times
Trump Effect' website takes credit for US investment made under Biden
Within hours of taking office in January, President Donald Trump boasted about attracting $3 trillion in new corporate investments to the United States. Since then, Trump has said the investments have swelled to $14 trillion, or roughly half of the nation's annual gross domestic product. The White House calls it "The Trump Effect" and features a rolling list on its website of more than 70 projects it says Trump's economic policies spurred, from a new bakery plant in Texas to a LEGO facility in Virginia and a microchip plant in Arizona. As of July 2, the website listed more than $2.6 trillion in U.S. investments, well short of the $14 trillion Trump boasts about. But a Reuters review found that just under half of the claimed spending on the website - totaling more than $1.3 trillion - originated under former President Joe Biden or represented routine spending repackaged to promote domestic investments. At least eight of the projects touted by the White House had sought or secured critical local incentive packages before Trump took office while at least a half dozen other projects had already been announced by local officials or the companies themselves, Reuters found. Two of the Trump Effect projects were aided by Biden's legislative efforts to boost domestic manufacturing, the review found. One company on the list, Swiss-based Roche, warned that Trump's plans to equalize U.S. and international drug prices now threatens its promised $50 billion in U.S. investments. Asked about taking credit for projects already underway before Trump came into office, the White House said the final investment decisions were announced under his watch and prove his economic policies are triggering U.S. investment. 'President Trump is the greatest closer in modern history, and his leadership and policies are a critical catalyst converting hypothetical discussions into firm investment commitments and ground being broken for new plants and offices," White House spokesman Kush Desai said. The Reuters review included interviewing local officials and reviewing public records and corporate statements. It was not clear in many cases what role, if any, Trump or his policies played in getting the deals across the line. Mark Zandi, the chief economist at Moody's Analytics, said his economic forecast - along with the consensus estimates - for investment in the economy has remained relatively unchanged despite the White House's claims of new historic investments. "I think despite all the announcements it hasn't translated into any change in expectations," Zandi said. "The fundamentals that ultimately drive investment spending, broadly, if anything, appear to have weakened since the start of the year." Trump's push to impose sweeping tariffs on dozens of trading partners has injected uncertainty into global markets, lowering economic projections and freezing investment decisions, Zandi said. Trump's supporters say his policies of deregulation combined with the extension of his corporate tax cuts last week have stoked interest from companies that will be converted into actual investments in the months ahead. "I think you're going to see a lot more investment later this year, and certainly into next year," Richard Stern, director of economic and budget policy at the conservative Heritage Foundation, said. THE TRUMP EFFECT The Trump Effect list is not exhaustive, according to the White House, and does not include the foreign deals the administration says Trump secured during his Middle East tour in May. The White House did not respond to a Reuters request to provide a breakdown of the $14 trillion in U.S. investments Trump claims he has attracted. Trump wouldn't be the first president to inflate or embellish economic activity on his watch. But the onetime businessman has made his dealmaking the centerpiece of his political persona, promising his presidency would ignite a manufacturing renaissance that would bring jobs back to the U.S. Some companies, largely in the pharmaceutical industry, repackaged existing spending that was later touted as new investment by Trump. The pharmaceutical companies also credited Trump's 2017 tax cuts for spurring domestic investment. Eli Lilly CEO David Ricks created the blueprint, said James Shin, a pharmaceutical analyst at Deutsche Bank Securities. Ricks joined top administration officials in February to announce $27 billion in new U.S. investments over five years. The figure drew praise from Trump who said it's evidence his tariffs were working to spur domestic manufacturing, but the figure represented a slight increase over the $23 million the company spent in the U.S. since 2020. "Everyone saw that Donald Trump gave David Ricks blessings every time he spoke," Shin said. "I think Lilly was quite shrewd in its timing and in its messaging." An Eli Lilly spokesperson did not address Reuters questions about the company's incremental increase in spending and what role Trump played, if any, in its announcement. CLAIMING CREDIT Here's a sampling of the projects included on the White House's Trump Effect website that Reuters found had been announced or were already in the pipeline prior to Trump's presidency: Hyundai: The South Korean carmaker was added to the Trump Effect list after announcing a $5.8 billion new Louisiana steel plant in March. But the company selected the Louisiana site in December 2024 after conducting a nationwide search, according to a state official. Hyundai did not respond to a request for comment, Corning: The global materials science company was added to the list after a $1.5 billion investment in Michigan was highlighted in an April press release. But the figure includes $900 million in funding announced in February of last year for the plant, and the project has benefited from federal tax credits under the CHIPS and Science Act, a bipartisan measure passed under Biden that incentivizes domestic production, the company confirmed to Reuters. Trump has called for Congress to claw back the chips funding, calling the legislation a "horrible, horrible thing." The company would not say whether Trump had any direct connection to the investment. LEGO: The iconic toy manufacturer announced a new $366 million distribution center in Virginia in May and was added to the Trump website. The company began working with Virginia on a package of state and local incentives in 2022, roughly three years before Trump took office, according to Pryor Green, the Virginia economic development spokesperson. The company did not respond to request for comment. Clasen Quality Chocolate: The candy company announced a new $230 million production facility in Virginia in February and was added to Trump's website. But the Virginia Economic Development Partnership began working with the company roughly seven months before Trump took office, and Governor Glenn Youngkin approved a $3 million grant for the project on December 3, Green told Reuters. The company did not respond to a request for comment. Chobani: The White House added the yogurt maker to its list after the company in April announced a $1.2 billion production plant in New York. But Chobani, which did not respond to a request for comment, had reached out to the state in May of last year about the project, state records show, and benefited from a state program that lures companies with shovel-ready sites. LOCAL INCENTIVES Some other deals touted by Trump were either struck before he took office, spurred by state and local incentive packages or represent routine capital investment, the Reuters review found. Pharmaceutical companies Merck and Johnson Johnson both announced billions of dollars in U.S. investments that included projects that were previously announced and already under construction, according to a review of company statements. The White House list included $2 billion in Merck projects already underway in North Carolina and Delaware and a $2 billion North Carolina project under construction by Johnson Johnson, statements show. Johnson Johnson did not respond to a request for comment, and Merck did not address Reuters' questions about Trump's role in its investment decisions. Many of the projects on the list relied heavily on state and local incentive packages - such as grants or tax breaks - that were approved prior to Trump taking office in January, Reuters found. States typically compete against one another for company investments, using incentive packages as bait. The locally-backed projects on the Trump Effect list include Diageo, a British alcoholic beverage company, whose $415 million new Alabama plant was aided by state and local tax incentives that date back to 2022, roughly three years before Trump took office, according to Stefania Jones, an Alabama Department of Commerce spokesperson. Diageo did not respond to requests for comment. Ireland-based power management company Eaton Corporation, French ceramics manufacturer Saint-Gobain and South Korean baker Paris Baguette, for example, were all highlighted on the Trump Effect list, but records and interviews show they all secured local incentive packages before Trump took office. The companies did not respond to requests for comment. TECH INVESTMENTS In some cases, the investment touted by the White House and the companies represented the normal cost of business. Apple CEO Tim Cook announced in February that his iconic company was going to invest $500 billion over five years to hire 20,000 workers and build new AI servers. Trump seized on the announcement, saying on his Truth Social media platform that it showed "faith in what we are doing." However, Apple's announced figure is in line with what one might expect the company to be spending anyway, given its financials, according to three analysts. "For Apple, most of this would have happened regardless of who's president," said Dan Ives, a senior equity analyst with Wedbush Securities. It also echoes previous commitments. Four years ago, a few months after Biden's inauguration, Apple announced an 'acceleration' of its U.S. investments, pledging to spend $430 billion and add 20,000 jobs over five years. In January 2018, during Trump's first term, the company said that its 'direct contribution to the U.S. economy' would be $350 billion over five years and that it planned to create 20,000 jobs over that period. Another pledge to spend $500 billion for new data centers to power artificial intelligence programs dubbed "Stargate" came from ChatGPT-maker OpenAI, Japanese conglomerate SoftBank and business software giant Oracle, whose executives joined Trump in the White House on his first full day as president in January to make the announcement. The companies said they planned to spend $100 billion 'immediately' but that they were still in negotiations with various states about where to place the new data centers. 'As announced in January, Stargate remains fully committed to investing up to $500 billion over the next four years to build AI infrastructure in the United States," SoftBank and OpenAI said in a joint statement to Reuters.
Economy ME
19 minutes ago
- Economy ME
AlphaGenome: How will Google DeepMind's AI model transform our understanding of the human genome?
Google DeepMind unveiled AlphaGenome, a groundbreaking artificial intelligence (AI) model poised to transform our understanding of the human genome and its impact on health, disease, and biotechnology. By leveraging state-of-the-art neural architectures and vast public genomic datasets, AlphaGenome delivers unprecedented insight into how genetic variants—both common and rare—affect gene regulation across the entire genome, not just the well-studied protein-coding regions that make up a mere 2 percent of our DNA. What is AlphaGenome? AlphaGenome is an advanced AI model developed by Google DeepMind, designed to predict how genetic variants impact gene regulation and other molecular processes at base-pair resolution across the entire genome. Unlike previous models that focused primarily on protein-coding DNA, AlphaGenome analyzes both coding and non-coding regions, offering a unified framework for interpreting the regulatory landscape of human genetics. Key highlights: Processes up to 1 million base pairs of DNA at once. Predicts thousands of molecular modalities, including gene expression, chromatin accessibility, RNA splicing, and protein binding. Integrates convolutional neural networks (CNNs) and transformers for both local motif detection and long-range genomic interactions. Trained on large-scale, multi-omic datasets (ENCODE, GTEx, 4D Nucleome, FANTOM5). Available via an API for non-commercial research, with plans for broader release. The need for advanced genomic AI The complexity of the human genome The human genome is a vast instruction manual, with over 3 billion DNA letters (base pairs). While only about 2 percent of these code for proteins, the remaining 98 percent—the non-coding regions—play crucial roles in regulating gene activity, determining when and where genes are turned on or off, and influencing susceptibility to diseases. Challenges in genomic interpretation: Variant effect prediction: Small changes (variants) in DNA can have profound or negligible effects, depending on their context. Non-coding regions: Most disease-associated variants identified by genome-wide association studies (GWAS) lie outside protein-coding regions, making their functional consequences difficult to interpret. Data volume: The scale and complexity of genomic data require models that can process long sequences and integrate diverse molecular signals. AlphaGenome was developed to address these challenges, providing a comprehensive, high-resolution view of how genetic variation shapes biology. Technical architecture of AlphaGenome Unified model for sequence-to-function prediction AlphaGenome's architecture is a hybrid neural network that combines the strengths of convolutional layers and transformer modules: Convolutional Neural Networks (CNNs): Detect short, local sequence motifs—such as transcription factor binding sites—by scanning DNA for recurring patterns. Transformers: Capture long-range dependencies and interactions between distant genomic elements, essential for modeling regulatory networks that span thousands of base pairs. This design enables AlphaGenome to analyze up to 1 million base pairs in a single pass, providing base-resolution predictions across vast genomic regions. Efficient training and inference Trained on Tensor Processing Units (TPUs), AlphaGenome achieves high computational efficiency, completing full model training in just four hours—using half the compute budget of its predecessor, Enformer. The model's architecture and data pipelines are optimized for both speed and accuracy, allowing rapid hypothesis generation and variant scoring at scale. Training data and benchmark performance Multi-omic datasets AlphaGenome's predictive power is rooted in its exposure to diverse, high-quality datasets: ENCODE: Comprehensive maps of functional elements in the genome. GTEx: Gene expression data across tissues. 4D Nucleome: Insights into genome structure and organization. FANTOM5: Transcriptional activity data. Benchmarking results Outperformed or matched specialized models in 24 out of 26 benchmark tests for variant effect prediction. Demonstrated superior performance in predicting regulatory effects, RNA splicing, and chromatin accessibility. Achieved state-of-the-art results in both single-sequence and variant effect prediction tasks. Key features and innovations Comprehensive variant effect prediction AlphaGenome can score both common and rare variants across the genome, including: Non-coding regulatory regions: Where most disease-associated variants reside. Protein-coding regions: Complementing tools like AlphaMissense. Multi-modal, base-resolution output Provides predictions for thousands of molecular properties at single-base resolution, enabling fine-grained analysis of genetic changes. Models RNA splice junctions directly—a critical advance for understanding diseases caused by splicing errors. Long-range genomic context Captures interactions between distant regulatory elements, such as enhancers and promoters, which are essential for accurate gene regulation modeling. Efficient, scalable, and accessible Trained efficiently on TPUs, with rapid inference capabilities. Available via API for non-commercial research, democratizing access for scientists worldwide. Applications in genomic research Decoding the non-coding genome AlphaGenome's ability to interpret the 98 percent of the genome that does not code for proteins opens new avenues for: Identifying regulatory variants that influence gene expression and disease risk. Prioritizing candidate variants in genome-wide association studies (GWAS). Understanding tissue-specific gene regulation and its disruption in disease. Functional genomics and hypothesis generation Researchers can use AlphaGenome to: Predict the impact of specific mutations before experimental validation. Generate functional hypotheses at scale, accelerating discovery in genetics and molecular biology. Impact on disease understanding and precision medicine From variant to function to disease AlphaGenome bridges the gap between genetic variation and biological function, providing insights that are crucial for: Rare disease diagnosis: Interpreting the effects of unique or de novo variants in patients with undiagnosed conditions. Cancer genomics: Understanding how somatic mutations in regulatory regions drive tumorigenesis. Pharmacogenomics: Predicting individual responses to drugs based on regulatory variants. Toward personalized medicine By enabling accurate prediction of variant effects across tissues and cell types, AlphaGenome supports the development of personalized therapies and precision diagnostics tailored to each individual's unique genetic makeup. Read more: UAE healthcare sector aims for 20 percent carbon emission reduction by 2030: Report Synthetic biology and beyond Designing synthetic DNA AlphaGenome's predictive capabilities extend to synthetic biology, where researchers aim to design custom DNA sequences with desired regulatory properties: Synthetic promoters and enhancers: Engineering regulatory elements for gene therapy or industrial biotechnology. Genome editing: Anticipating the consequences of CRISPR and other genome-editing interventions. Expanding to other species DeepMind has indicated plans to extend AlphaGenome's framework to new species , facilitating comparative genomics and cross-species functional annotation. AlphaGenome vs. previous models Feature AlphaGenome Enformer (2022) AlphaMissense (2023) Sequence length Up to 1 million bp Up to 200,000 bp N/A (missense focus) Coding & non-coding regions Yes Yes Coding only Variant effect prediction Yes (all regions) Limited Missense only Multi-modal output Thousands of types Dozens Protein function Splice junction modeling Direct Indirect No Training efficiency 4 hours on TPUs 8+ hours N/A Benchmark performance 24/26 top scores 18/26 N/A AlphaGenome represents a substantial leap in both scale and accuracy compared to previous models, especially in non-coding variant interpretation and multi-modal prediction. Ethical, societal, and clinical considerations Interpretability and trust As AI models become central to genomic interpretation, issues of transparency, explainability, and clinical validation are paramount. AlphaGenome's predictions must be interpreted within the context of experimental evidence and patient care, with careful attention to: False positives/negatives in variant effect prediction. Equity and access to advanced genomic tools across different populations and healthcare systems. Data privacy and security Handling genomic data raises significant privacy concerns, necessitating robust safeguards for patient information and compliance with global regulations. The human element As noted by AI alignment researchers, the psychological and informational context in which genomic insights are delivered is as important as their technical accuracy. AI must support clinicians in providing clear, compassionate communication to patients. The road ahead: Future developments Clinical integration DeepMind plans to extend AlphaGenome for clinical applications, including fine-tuning for disease-specific tasks, integration with electronic health records, and support for clinical decision-making. Expansion to other organisms and modalities Ongoing work aims to adapt AlphaGenome for other species and new molecular phenotypes, broadening its impact across biology and medicine. Open science and collaboration By making AlphaGenome available via API for non-commercial research, DeepMind promotes global collaboration and accelerates discovery in genomics. Final word AlphaGenome marks a new era in computational genomics, offering a unified, scalable, and accurate framework for interpreting the functional consequences of genetic variation across the entire genome. Its release in 2025 represents a milestone not just for AI and genomics, but for the broader quest to understand the language of life and harness it for human health, disease prevention, and biotechnological innovation.
Zawya
an hour ago
- Zawya
US consumers happier about finances, expect stable inflation, New York Fed says
Americans' outlook on inflation was little changed last month as households upgraded their views on the state of their finances and ability to get credit, according to a report released on Tuesday by the New York Federal Reserve. As of June, inflation one year from now was expected to be 3%, down from the expected 3.2% in May, while the outlooks at the three- and five-year-ahead horizons were unchanged at 3% and 2.6%, respectively, according to the latest New York Fed Survey of Consumer Expectations. Amid the calm outlook for future price increases, the survey found that respondents had "markedly" upgraded their assessment of their personal financial situation relative to last year, while noting credit had grown easier to access. Respondents also upgraded their expectations about the state of their financial situations a year from now. The survey found mixed expectations for future earnings and income in June, while the outlook for employment improved. Although the New York Fed found in its poll that the public's outlook for inflation was little changed last month, households projected in June an acceleration in year-ahead gains in the cost of gasoline, medical care, college and rent, while the expected rise in food costs held steady relative to May. Near-term inflation expectations recorded by the New York Fed have been volatile this year as President Donald Trump launched an aggressive trade war against many U.S. trading partners. The president's trade agenda, which features the imposition of high tariffs on imported goods, is widely expected to push up inflation and depress growth and hiring. Those import levies helped drive up near-term expected inflation, and as the president appears to have capitulated so far on the most draconian of his levies, worries about higher inflation have eased. Other surveys like the University of Michigan report on consumer sentiment have also shown reduced worries about future inflation. Meanwhile, long-term inflation expectations have remained mostly stable, which is good news for Fed officials, who believe that development suggests confidence that over the long run inflation will not be a major concern. Fed officials, however, are expecting higher inflation this year due to the tariffs, which they expect to wane starting next year. Fed officials penciled in two rate cuts for this year at their policy meeting last month but offered little guidance as to when that might happen. Some Fed officials were eyeing the July 29-30 policy meeting as a good time for a rate cut, but solid job market data for June appears to have taken that idea off the board. In comments after the June 17-18 meeting, Fed Chair Jerome Powell said "our obligation is to keep longer-term inflation expectations well-anchored and to prevent a one-time increase in the price level from becoming an ongoing inflation problem." (Reporting by Michael S. Derby; Editing by Paul Simao)
