minus-squarewingiee@lemm.eetoTechnology@lemmy.world•Cloudflare announces AI Labyrinth, which uses AI-generated content to confuse and waste the resources of AI Crawlers and bots that ignore “no crawl” directives.linkfedilinkEnglisharrow-up17·8 hours agoI assume a crawler which adheres to robots.txt linkfedilink
I assume a crawler which adheres to robots.txt