The open web is closing down for unwanted automated traffic

A growing number of websites are taking steps to ban AI bot traffic so that their work isn't used as training data and their servers aren't overwhelmed by non-human users. However, some companies are ignoring the bans and scraping anyway.

Online traffic analysis conducted by BuiltWith, a web metrics biz, indicates that the number of publishers trying to prevent AI bots from scraping content for use in model training has surged since July.

About 5.6 million websites presently have added OpenAI's GPTBot to the disallow list in their robots.txt file, up from about 3.3 million at the start of July 2025. That's an increase of almost 70 percent.

...read more at theregister.com