@[email protected]cake to Lemmy [email protected] • 1 year agoChad scrapersh.itjust.worksimagemessage-square96fedilinkarrow-up11.06Karrow-down129
arrow-up11.03Karrow-down1imageChad scrapersh.itjust.works@[email protected]cake to Lemmy [email protected] • 1 year agomessage-square96fedilink
minus-square@[email protected]linkfedilink73•1 year agoJust a heads up for companies thinking it’s wrong to scrap: if you don’t want info to be scraped, don’t put it on the internet.
Just a heads up for companies thinking it’s wrong to scrap: if you don’t want info to be scraped, don’t put it on the internet.
But, but, norobots.txt!
Chad doesn’t care!
uses curl manually instead