• mox@lemmy.sdf.org
    link
    fedilink
    arrow-up
    7
    ·
    7 months ago

    Of course, nothing stops a bot from picking a user agent field that exactly matches a web browser.

    • JackbyDev
      link
      fedilink
      English
      arrow-up
      4
      arrow-down
      1
      ·
      7 months ago

      Nothing stops a bot from choosing to not read robots.txt