But AI crawlers have been particularly ill-behaved over the past two years as the generative AI boom has played out. OpenAI in August 2023 made it known its web crawlers would respect robots.txt files ...