The scrapers are not actually an ai, they are just dumb scrapers there to get as much textual information as possible.

If they have to do Anubis tests, that is going to take more time to get the data they scrape. I suspect that they are probably paid per page they provide, so more time per page is less money for them.