Files Crawlers Depend On: robots, sitemaps, llms, favicons

Practical guide to monitoring robots.txt, sitemap.xml, llms.txt, favicons, and social metadata. Catch 404s, bad directives, cache bugs, and deploys that quietly break discovery.