Cloudflare Implements New Rules to Manage AI Bots

Sep 15, 2026

73 min

Sep 15, 2026 07:31

The rules of the web change today as Cloudflare cracks down on AI bots

## Cloudflare's New Policy

Starting September 15, Cloudflare is implementing new rules to manage how AI bots interact with websites. This change aims to give website owners more control over their content by distinguishing between different types of automated traffic.

## Categories of AI Bots

Cloudflare now categorizes AI bots into three groups: search crawlers, training bots, and agent bots. Search crawlers, which index web content for search engines, will continue to be allowed by default. However, bots used for AI training or as agents will be blocked on pages with advertisements unless website owners choose otherwise.

## Impact on Publishers and AI Companies

This policy shift could significantly affect publishers and AI companies. Publishers are concerned about AI systems using their content without compensation, while AI companies may face restrictions on accessing data for training models. Cloudflare's move reflects a growing tension over content usage rights in the digital landscape.

## Implementation Details

The new rules apply to new domains joining Cloudflare and existing free-tier customers who haven't updated their settings by the deadline. Website owners can adjust these settings through their Cloudflare dashboard. Notably, crawlers performing multiple functions will be subject to the most restrictive setting applicable.

## Significance of the Change

This development highlights the evolving relationship between the open web and AI technologies. By providing analytics on AI bot activity, Cloudflare aims to help website owners make informed decisions about allowing or blocking different types of automated traffic.

Read the full story at the source

What you need to know to get Emirates ID?

Leave your details and get a guide as a gift to avoid mistakes

Article contents

LinkedIn X Facebook WhatsApp Telegram