# Search crawlers: everything is crawlable, and that is on purpose. # # The pages that should stay out of search carry their own noindex meta: # fod/, login.html, index-full.html and schedule/print.html. There are # deliberately no Disallow rules for them, because Disallow blocks the crawl, # and a page that is never crawled is never read, so its noindex never takes # effect. Those URLs can then still surface from an external link, just with no # description under them. Letting the crawler in is what gets them dropped. # # The AI rules below do NOT contradict that. The reasoning above is about # search indexing, where a Disallow is self-defeating because it suppresses the # very fetch that would have read the noindex. Training crawlers index nothing # and there is no noindex for them to read, so a Disallow is the only signal # available and it costs nothing. # # The split is deliberate: # - Training crawlers are disallowed. They ingest the work to reproduce it. # - AI SEARCH crawlers are allowed. They cite and link back, which is how a # reader looking for this schedule actually finds it. # # None of this is a technical barrier. robots.txt is advisory, it is honoured # by the well-behaved and ignored by everyone else, and it is not in the path # at all when a person pastes a URL into a chat window. It is here to state the # terms, so that reuse cannot later be called an honest misunderstanding. # # The site's content is not licensed for reuse. See /LICENSE and /llms.txt. # ─── Training crawlers: not permitted ─────────────────────────────── User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: CCBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Meta-ExternalAgent Disallow: / User-agent: FacebookBot Disallow: / User-agent: Bytespider Disallow: / User-agent: Amazonbot Disallow: / User-agent: cohere-ai Disallow: / User-agent: Diffbot Disallow: / User-agent: Omgilibot Disallow: / User-agent: AI2Bot Disallow: / User-agent: Timpibot Disallow: / User-agent: ImagesiftBot Disallow: / # ─── Everything else, including AI search and ordinary search ─────── # # Explicitly welcome, and named here so the intent is unmistakable rather than # merely inherited from the wildcard below: OAI-SearchBot, ChatGPT-User, # Claude-SearchBot, Claude-User, PerplexityBot, Googlebot, Bingbot, # DuckDuckBot, Applebot. User-agent: * Allow: / Sitemap: https://greateststlv.com/sitemap.xml