Cynicus Rex@lemmy.ml to Privacy@lemmy.mlEnglish · 10 个月前How to block AI Crawler Bots using robots.txt filewww.cyberciti.bizexternal-linkmessage-square62linkfedilinkarrow-up1110arrow-down132
arrow-up178arrow-down1external-linkHow to block AI Crawler Bots using robots.txt filewww.cyberciti.bizCynicus Rex@lemmy.ml to Privacy@lemmy.mlEnglish · 10 个月前message-square62linkfedilink
minus-squareasudox@lemmy.worldlinkfedilinkarrow-up6arrow-down1·10 个月前Not sure if that is effective at all. Why would a crawler check the robots.txt if it’s programmed to ignore it anyways?
minus-squareɐɥO@lemmy.ohaa.xyzlinkfedilinkarrow-up16·10 个月前cause many crawlers seem to explicitly crawl “forbidden” sites
minus-squareCrashumbc@lemmy.worldlinkfedilinkEnglisharrow-up3·10 个月前Google and script kiddies copying code…
minus-squareMangoPenguin@lemmy.blahaj.zonelinkfedilinkEnglisharrow-up1·9 个月前You could also place the same page as a hidden link on your home page.
Not sure if that is effective at all. Why would a crawler check the robots.txt if it’s programmed to ignore it anyways?
cause many crawlers seem to explicitly crawl “forbidden” sites
Google and script kiddies copying code…
You could also place the same page as a hidden link on your home page.