Agent-Friendly Website Directory

A database that tracks which websites allow AI agent access and which block them (and why). Helps developers building agent/automation systems know upfront which platforms their agents can interact with, reducing wasted execution time and errors.

The robots.txt conversation has gotten genuinely complicated as LLMs and autonomous agents collide with platforms trying to protect their data moats — Cloudflare's AI bot blocking, Reddit's API crackdown, and LinkedIn's aggressive scraping lawsuits have all landed in the past 18 months, so developers are flying blind on what's actually accessible. No clear incumbent owns this space, though BuiltWith and similar fingerprinting directories are adjacent substitutes that don't address the agent-access angle. The $1k–5k/mo revenue band is realistic only if monetization is through API access or a paid tier for freshness and bulk queries, since a static free directory has no obvious paywall; the ceiling is low because the core data ages fast and the audience is narrow. The biggest risk is that the information decays too quickly to maintain credibly — a site's agent policy can change overnight, and without automated monitoring infrastructure the directory becomes stale misinformation faster than any small team can manually update it.