# ============================================================ # GoTax robots.txt # Reviewed: 1-June-2026 # ============================================================ # ------------------------------------------------------------ # DEFAULT RULES - applies to all crawlers not named below # (search engines, Googlebot, Bingbot, etc.) # ------------------------------------------------------------ User-agent: * Allow: / Disallow: /admin Disallow: /cgi-bin Disallow: /js Disallow: /no-script Disallow: /site/login Disallow: /login Disallow: /account Disallow: /checkout Disallow: /app # ------------------------------------------------------------ # AI SEARCH / RETRIEVAL / USER CRAWLERS - ALLOWED # These power citations and answers in ChatGPT search, Claude, # Perplexity and user-initiated lookups. Keep these allowed to # support AI retrievability (AEO / GEO). They do not train models. # Private and transactional paths are still excluded. # ------------------------------------------------------------ User-agent: OAI-SearchBot Allow: / Disallow: /admin Disallow: /cgi-bin Disallow: /js Disallow: /no-script Disallow: /site/login Disallow: /login Disallow: /account Disallow: /checkout Disallow: /app User-agent: ChatGPT-User Allow: / Disallow: /admin Disallow: /cgi-bin Disallow: /js Disallow: /no-script Disallow: /site/login Disallow: /login Disallow: /account Disallow: /checkout Disallow: /app User-agent: Claude-SearchBot Allow: / Disallow: /admin Disallow: /cgi-bin Disallow: /js Disallow: /no-script Disallow: /site/login Disallow: /login Disallow: /account Disallow: /checkout Disallow: /app User-agent: Claude-User Allow: / Disallow: /admin Disallow: /cgi-bin Disallow: /js Disallow: /no-script Disallow: /site/login Disallow: /login Disallow: /account Disallow: /checkout Disallow: /app User-agent: PerplexityBot Allow: / Disallow: /admin Disallow: /cgi-bin Disallow: /js Disallow: /no-script Disallow: /site/login Disallow: /login Disallow: /account Disallow: /checkout Disallow: /app User-agent: Perplexity-User Allow: / Disallow: /admin Disallow: /cgi-bin Disallow: /js Disallow: /no-script Disallow: /site/login Disallow: /login Disallow: /account Disallow: /checkout Disallow: /app # ------------------------------------------------------------ # AI TRAINING CRAWLERS - DISALLOWED # Your published policy (llms.txt and ai.txt) states # Training-Allowed: no. GPTBot and ClaudeBot collect data for # model training, so they are blocked to match that policy. # Blocking GPTBot does NOT remove you from ChatGPT search, # because OAI-SearchBot (allowed above) handles that. # ------------------------------------------------------------ User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / # ------------------------------------------------------------ # GOOGLE-EXTENDED - DECISION REQUIRED (see implementation note) # Google bundles Gemini training AND Gemini grounding under this # one token. Your AI Overviews presence runs off the normal # Googlebot index and is NOT affected by this setting. # # OPTION A (honours no-training policy): block it. # Uncomment the two lines below. # OPTION B (maximises Gemini grounding visibility): leave it # allowed by keeping the lines commented out. # User-agent: Google-Extended Disallow: / # ------------------------------------------------------------ # ------------------------------------------------------------ # AI policy references (human-readable only). # These are NOT honoured as robots.txt directives by crawlers, # but the files themselves are live and worth keeping at root. # AI policy: https://www.gotax.com.au/ai.txt # LLM guide: https://www.gotax.com.au/llms.txt # ------------------------------------------------------------ Sitemap: https://www.gotax.com.au/sitemap.xml Sitemap: https://www.gotax.com.au/blog_sitemap.xml Sitemap: https://business.gotax.com.au/sitemap.xml