# robots.txt for www.intime.de # Default: allow all reputable crawlers. Block dev/test/internal paths. # AI policy: allow retrieval/answer-engine bots, block bulk training scrapers. # Change log: see docs/adr/ai-crawler-policy.md User-agent: * Allow: / Disallow: /dev/ Disallow: /api/ Disallow: /hidden-pages/ Disallow: /weitere-navigationsleisten/ Disallow: /translate-to-english-schulungsseiten/ # --- AI / LLM crawlers: training bots (blocked) --------------------------- User-agent: GPTBot Disallow: / User-agent: CCBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Google-Extended Disallow: / User-agent: Bytespider Disallow: / User-agent: FacebookBot Disallow: / User-agent: Amazonbot Disallow: / User-agent: Meta-ExternalAgent Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: PerplexityBot Disallow: / # --- AI answer-engine / retrieval bots (explicitly allowed) -------------- # These fetch pages to cite in real-time AI answers. We want visibility here. User-agent: OAI-SearchBot Allow: / Disallow: /dev/ Disallow: /api/ User-agent: ChatGPT-User Allow: / Disallow: /dev/ Disallow: /api/ User-agent: PerplexityBot-User Allow: / Disallow: /dev/ Disallow: /api/ Sitemap: https://www.intime.de/sitemap-index.xml