Set robots.txt for GHL Domains
Purpose
Standardize the robots.txt served by GHL-hosted domains so search engines can crawl production pages. Applies to the SC website and client websites. Every GHL-based client (and SC itself) has a go.[domain] subdomain that serves GHL funnel/website pages; when that subdomain has no robots.txt entered, crawlers are unable to crawl it and the pages do not get indexed. The standard fix is to enter a permissive robots.txt (allow-all + sitemap).
Scope
Covers the GHL Domains & URL Redirects → Edit Domain → "Robots.txt code" field. The primary case is production go.[domain] subdomains. The staging block-all case is covered as an exception (see Exceptional circumstances).
Background
The "Robots.txt code" field on a GHL domain is empty by default. An empty/missing robots.txt on a go.[domain] subdomain prevents crawlers from crawling the pages served there. Entering an allow-all robots.txt with a Sitemap: directive resolves this and points crawlers at the sitemap.
Prerequisites
- GHL admin access to the target location
- The
go.[domain]subdomain connected under Settings → Domains & URL Redirects - The subdomain's sitemap URL (typically
https://go.[domain]/sitemap.xml) - Familiarity with robots.txt syntax
Procedure — production go subdomain (standard)
-
In GHL, go to Settings → Domains & URL Redirects → Domains tab.
-
Locate the
go.[domain]row, open its ⋮ menu, and click Edit. The Edit Domain dialog opens. -
In the Robots.txt code box, enter the following (replace
[domain]):User-agent: *
Allow: /
Sitemap: https://go.[domain]/sitemap.xmlThe
Sitemap:line must point to the subdomain's real sitemap. -
Click Save.
Verification
- Visit
https://go.[domain]/robots.txtand confirm it shows the allow-all rule and theSitemap:line. - Confirm the
Sitemap:URL itself returns HTTP 200 in a browser — verify it actually resolves before relying on it; do not assume GHL auto-generates one. - Optionally, run Search Console → URL Inspection (or the robots.txt tester) on a page under the subdomain to confirm it is crawlable.
Exceptional circumstances — staging block-all
When intentionally hiding a staging or pre-launch subdomain from search engines, block all crawlers instead:
User-agent: *
Disallow: /
⚠️ Mandatory go-live step: before publishing to the live domain, remove the block-all and replace it with the production allow-all + sitemap from the standard procedure above. A Disallow: / that leaks onto a live domain silently prevents indexation. Tie this to the go-live checklist so it is never skipped.
Troubleshooting
robots.txt not accessible
- Wait up to 10 minutes for propagation.
- Re-check the domain configuration in Domains & URL Redirects.
Pages still not crawled / not indexed
- Confirm there is no residual
Disallow: /in the Robots.txt code field (most common cause after a staging-to-live transition). - Check Search Console for crawl/coverage errors and request indexing if needed.
Related Documents
- Make a GHL Funnel Page Indexable on WordPress
- Rank Math Configuration Standard — robots.txt on a WordPress domain (owned by Rank Math, not this SOP)
- Client Technical SEO Audit SOP — recurring robots, sitemap and indexability checks
- GHL B-Series Funnel Activation SOP
- Client Onboarding Data Fields
Revision History
| Date | Version | Change Description | Author |
|---|---|---|---|
| 2025-08-07 | 1.0 | Initial staging SOP | Symphony Core Systems Team |
| 2026-06-14 | 2.0 | Generalized to production go subdomains; allow-crawl now standard, staging block-all demoted to exception | Symphony Core Systems Team |
| 2026-10-31 | 2.1 | Related links to the archived SearchAtlas SOPs replaced with the Rank Math standard (WordPress robots.txt owner) and the technical SEO audit SOP | Symphony Core Systems Team |