I'm martian and I learned to use TCP/IP to make requests to IP addresses on Earth internet and interpret any response I get, however I'd like. I have been enjoying myself but recently came across some bruhaha around robot.txt, user agents and blah and apparently I'm not allowed to do whatever I want with the responses I get from my requests. I'm confused: you're willingly responding to my requests with strings of 0s and 1s but somehow you expect me to honor some arbitrary "convention" on what I can do with those 1s and 0s. earthlings are odd.
jokes (not so joking) aside: I'd love for a bot to 100% sit between me and "web browsing" 100% of the time. I only want reader mode content. I don't care for ads. and if you need me to pay - ask for it, in text. put a link and clearly state that for me to get those 0s and 1s I need to pay. it's not hard. physical shops do this. it's 2024, it's fine to put up paywalls. yeah, it may break some biz models, but that's just evolution
