robots.txt Analyzer

Parse and audit any site's robots.txt in one call: user-agent groups, allow/disallow rules, declared sitemaps, and crawl-block issues like…

What counts as one robots.txt scan

One site robots.txt parsed: user-agent groups, rules, sitemaps, and issues.

Call it from code

curl -X POST \
  "https://api.apify.com/v2/acts/eliai~robots-txt-analyzer/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d @input.json

The input fields are on the store listing. Results come back as JSON rows.

Price $0.02 per robots.txt scan Apify free-plan rate, read Oct 23, 1:24 PM ET. Billed by Apify; paid plans pay less. The listing is the final word. Open on the Apify Store

More SEO tools

Core Web Vitals & PageSpeed Insights CheckerBulk Core Web Vitals and PageSpeed Insights reports from Google's official API: performance score, lab LCP, CLS, TBT, FCP and Speed Index…$0.01 per vitals report
Broken Link CheckerBroken link checker: give it a start URL, it checks every link on the page and returns the broken ones as JSON — link URL and HTTP status…$0.02 per page check
SEO Audit ToolSEO audit for any URL or whole site: crawl mode finds redirect chains, broken links, duplicate titles; page mode checks title/meta…$0.10 per audited page