Our example was simpler because it was not crawling but scraping a specific page.