pull down to refresh
102 sats \ 0 replies \ @beameduplol 8h \ on: Crawling a billion web pages in just over 24 hours, in 2025 tech
There is a tool I read up online to mitigate crawlers/scrapers, it's called uncaptcha, FOSS (if I am not wrong) and the way it works is makes browsers do a small proof-of-work to access websites. Usually takes 4 - 6 seconds on a normal computer, but adds magnitudes of delay to crawlers/scrapers