I think it could be potentially easier to thwart malicious bots than “honest” bots. I figure a bot that doesn’t care about robots.txt and whatnot would try to gobble up as many pages as it could find. You could easily place links into HTML that aren’t visible to regular users and a “greedy” bot would follow it anyway. From there you could probably have a website within a website that’s being generated by AI on the fly. To keep the bots from running up your bills you probably want it to be mostly static.