Robots Rule Tester

Test URLs against robots.txt with real Google semantics: longest-match, Allow wins ties, UA group selection (test Googlebot, GPTBot…

What counts as one tested URL

One URL tested against its site's robots.txt with Google longest-match semantics (matched rule, UA group, sitemaps, crawl-delay). Invalid URLs are recorded free.

Call it from code

curl -X POST \
  "https://api.apify.com/v2/acts/eliai~robots-rule-tester/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d @input.json

The input fields are on the store listing. Results come back as JSON rows.

Price $0.0008 per tested URL Apify free-plan rate, read Oct 23, 1:24 PM ET. Billed by Apify; paid plans pay less. The listing is the final word. Open on the Apify Store

More SEO tools

Core Web Vitals & PageSpeed Insights CheckerBulk Core Web Vitals and PageSpeed Insights reports from Google's official API: performance score, lab LCP, CLS, TBT, FCP and Speed Index…$0.01 per vitals report
Broken Link CheckerBroken link checker: give it a start URL, it checks every link on the page and returns the broken ones as JSON — link URL and HTTP status…$0.02 per page check
robots.txt AnalyzerParse and audit any site's robots.txt in one call: user-agent groups, allow/disallow rules, declared sitemaps, and crawl-block issues like…$0.02 per robots.txt scan