alyaza [they/she]@beehaw.orgM to Technology@beehaw.orgEnglish · 9 months agoThe rise and fall of robots.txt: As unscrupulous AI companies seek out more and more data, the basic social contract of the web is falling apart.www.theverge.comexternal-linkmessage-square7fedilinkarrow-up1160arrow-down10cross-posted to: technology@midwest.socialwolnyinternet@szmer.infotechnology@lemmy.world
arrow-up1160arrow-down1external-linkThe rise and fall of robots.txt: As unscrupulous AI companies seek out more and more data, the basic social contract of the web is falling apart.www.theverge.comalyaza [they/she]@beehaw.orgM to Technology@beehaw.orgEnglish · 9 months agomessage-square7fedilinkcross-posted to: technology@midwest.socialwolnyinternet@szmer.infotechnology@lemmy.world
minus-squaregayhitler420@lemm.eelinkfedilinkarrow-up61·9 months agorobots.txt isn’t a basic social contract, it’s a file intended to save web crawlers precious resources.
robots.txt isn’t a basic social contract, it’s a file intended to save web crawlers precious resources.