Check robots.txt file of a website for errors and typos
GitHub - nickmoline/robots-checker: Library to check a url for all possible robots exclusion protocols (robots.txt, http X-Robots-Tag header, meta robots tag, or http status codes)
Robots.txt: What You Need to Know - LinkBuilding HQ