Tea to Technology@lemmy.worldEnglish · 10 hours agoCloudflare announces AI Labyrinth, which uses AI-generated content to confuse and waste the resources of AI Crawlers and bots that ignore “no crawl” directives.blog.cloudflare.comexternal-linkmessage-square105fedilinkarrow-up1698arrow-down15cross-posted to: [email protected][email protected]
arrow-up1693arrow-down1external-linkCloudflare announces AI Labyrinth, which uses AI-generated content to confuse and waste the resources of AI Crawlers and bots that ignore “no crawl” directives.blog.cloudflare.comTea to Technology@lemmy.worldEnglish · 10 hours agomessage-square105fedilinkcross-posted to: [email protected][email protected]
minus-squarewingiee@lemm.eelinkfedilinkEnglisharrow-up18·9 hours agoI assume a crawler which adheres to robots.txt
minus-squarelily33@lemm.eelinkfedilinkEnglisharrow-up4·8 hours agoI would love to think so. But the word “verified” suggests more.
minus-squarekilleronthecorner@lemmy.worldlinkfedilinkEnglisharrow-up1·5 hours agoIP verification is a not uncommon method for commercial crawlers
I assume a crawler which adheres to robots.txt
I would love to think so. But the word “verified” suggests more.
IP verification is a not uncommon method for commercial crawlers