• Cosmic Cleric@lemmy.world
    link
    fedilink
    English
    arrow-up
    6
    arrow-down
    1
    ·
    9 months ago

    When sites regularly have a robots.txt that says “only Google may crawl”

    Is that actually true?

    If so, why would they do that?