Not sure if throwing resource-wasting technology at a resource-wasting problem is a cool idea on an already burning planet …

This is amazing. The number of crawlers that don’t respect crawling directives is out of hand. Feed them garbage till they can’t pay their bills.

On the one hand this is good news, because by allowing stealing data from websites we would just financially kill those who produce legitimate data (e.g. news sites having articles stolen without receiving ad revenue as only bots visit them).

On the other hand - AI tools will get dumber, which makes me sad because I personally hoped for having one day a supercharged assistant that reads and summarises Internet for me. Tools like Perplexity creating research on given topics looked very promising