Skip to main content

Set robots.txt for GHL Domains

Purpose​

Standardize the robots.txt served by GHL-hosted domains so search engines can crawl production pages. Applies to the SC website and client websites. Every GHL-based client (and SC itself) has a go.[domain] subdomain that serves GHL funnel/website pages; when that subdomain has no robots.txt entered, crawlers are unable to crawl it and the pages do not get indexed. The standard fix is to enter a permissive robots.txt (allow-all + sitemap).

Scope​

Covers the GHL Domains & URL Redirects → Edit Domain → "Robots.txt code" field. The primary case is production go.[domain] subdomains. The staging block-all case is covered as an exception (see Exceptional circumstances).

Background​

The "Robots.txt code" field on a GHL domain is empty by default. An empty/missing robots.txt on a go.[domain] subdomain prevents crawlers from crawling the pages served there. Entering an allow-all robots.txt with a Sitemap: directive resolves this and points crawlers at the sitemap.

Prerequisites​

  • GHL admin access to the target location
  • The go.[domain] subdomain connected under Settings → Domains & URL Redirects
  • The subdomain's sitemap URL (typically https://go.[domain]/sitemap.xml)
  • Familiarity with robots.txt syntax

Procedure — production go subdomain (standard)​

  1. In GHL, go to Settings → Domains & URL Redirects → Domains tab.

  2. Locate the go.[domain] row, open its ⋮ menu, and click Edit. The Edit Domain dialog opens.

  3. In the Robots.txt code box, enter the following (replace [domain]):

    User-agent: *
    Allow: /

    Sitemap: https://go.[domain]/sitemap.xml

    The Sitemap: line must point to the subdomain's real sitemap.

  4. Click Save.

Verification​

  1. Visit https://go.[domain]/robots.txt and confirm it shows the allow-all rule and the Sitemap: line.
  2. Confirm the Sitemap: URL itself returns HTTP 200 in a browser — verify it actually resolves before relying on it; do not assume GHL auto-generates one.
  3. Optionally, run Search Console → URL Inspection (or the robots.txt tester) on a page under the subdomain to confirm it is crawlable.

Exceptional circumstances — staging block-all​

When intentionally hiding a staging or pre-launch subdomain from search engines, block all crawlers instead:

User-agent: *
Disallow: /

⚠️ Mandatory go-live step: before publishing to the live domain, remove the block-all and replace it with the production allow-all + sitemap from the standard procedure above. A Disallow: / that leaks onto a live domain silently prevents indexation. Tie this to the go-live checklist so it is never skipped.

Troubleshooting​

robots.txt not accessible​

  • Wait up to 10 minutes for propagation.
  • Re-check the domain configuration in Domains & URL Redirects.

Pages still not crawled / not indexed​

  • Confirm there is no residual Disallow: / in the Robots.txt code field (most common cause after a staging-to-live transition).
  • Check Search Console for crawl/coverage errors and request indexing if needed.

Revision History​

DateVersionChange DescriptionAuthor
2025-08-071.0Initial staging SOPSymphony Core Systems Team
2026-06-142.0Generalized to production go subdomains; allow-crawl now standard, staging block-all demoted to exceptionSymphony Core Systems Team
2026-10-312.1Related links to the archived SearchAtlas SOPs replaced with the Rank Math standard (WordPress robots.txt owner) and the technical SEO audit SOPSymphony Core Systems Team