Parsers for robots.txt (aka Robots Exclusion Standard / Robots Exclusion Protocol), Robots Meta Tag, and X-Robots-Tag
robots-txt
robots
meta-tags
robots-parser
robots-exclusion-standard
robotstxt
robots-tag
robots-exclusion-protocol
x-robots-tag
-
Updated
May 2, 2024 - C#