Migrate to archive.sparkpost.com with host-conditional noindex#855
Merged
Conversation
Moves the docs site from support.sparkpost.com to archive.sparkpost.com so the content can be archived (and later 301'd to bird.com) without competing with bird.com in search. Adds noindex/nofollow at three layers (robots.txt, <meta name="robots">, X-Robots-Tag header) so search engines drop the archive. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
✅ Deploy Preview for support-docs ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
rafaelmatsumotomb
approved these changes
Jun 25, 2026
viniciusgiles
approved these changes
Jun 25, 2026
balupillai
approved these changes
Jun 25, 2026
The base-URL + noindex change in this branch applies to every hostname this deploy serves. During the gap between this PR landing and the CloudFront redirect distribution going live for support.sparkpost.com, that's incorrect for SEO: archive.sparkpost.com SHOULD be noindex (it's the archived copy), but support.sparkpost.com SHOULD still be indexed so its existing link equity is preserved until CloudFront can 301 it to bird.com counterparts. If support is noindex'd during the gap, Google can't see the eventual 301s (robots.txt Disallow blocks re-crawl), so equity that would otherwise transfer to bird.com is lost instead. The longer the gap, the more decays. Add a Netlify Edge Function that runs on every request and, for requests with Host: support.sparkpost.com: - serves a permissive /robots.txt (no Disallow:/) - strips X-Robots-Tag from response headers - strips the <meta name="robots" content="noindex,..."> tag from HTML archive.sparkpost.com and all other hosts (deploy previews, *.netlify.app) pass through unmodified — noindex stays in place. REMOVE this function (and the netlify.toml comment pointing at it) once the CloudFront cutover is complete. After that, support.sparkpost.com no longer hits Netlify, so the hostname-conditional logic becomes dead code. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PauloJeunonSousa
changed the title
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 56fa499. Configure here.
Discovered via local netlify dev testing: `response.headers.delete('X-Robots-Tag')`
is silently ignored when the header is set in netlify.toml's [[headers]] block.
Netlify re-applies the netlify.toml headers after the edge function returns,
overriding any deletions — but it respects values the function explicitly sets.
Switch from delete() to set('X-Robots-Tag', 'all'). Google treats X-Robots-Tag:
all as equivalent to no header (canonical "ignore any prior noindex; index
normally"), so the effect is identical to the intended deletion. Other header
sets (X-Edge-Probe sentinel during testing) confirmed mutations survive — only
deletes of netlify.toml-sourced headers are eaten.
Verified locally with netlify dev across the full host × content-type matrix:
- archive.sparkpost.com HTML: X-Robots-Tag: noindex, nofollow + meta intact ✓
- support.sparkpost.com HTML: X-Robots-Tag: all + meta stripped ✓
- archive.sparkpost.com /robots.txt: Disallow: / served as built ✓
- support.sparkpost.com /robots.txt: permissive (User-agent: *) ✓
- support.sparkpost.com JSON: X-Robots-Tag: all + body untouched ✓
- archive.sparkpost.com JSON: X-Robots-Tag: noindex + body untouched ✓
- Deploy preview host (deploy-preview-*.netlify.app): noindex preserved ✓
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
rafaelmatsumotomb
approved these changes
Jun 26, 2026
…m tsc
Two issues flagged in the PR:
1. Cursor Bugbot (high severity): after rewriting an HTML response body in
the edge function for support.sparkpost.com, only `content-length` was
removed from the copied headers. If the origin's response was gzip- or
brotli-encoded (which Netlify's CDN does opportunistically), the
`response.text()` call decoded the body to plain text but the stale
`Content-Encoding` header survived. Clients would then try to decompress
plain text and fail. Fix: also delete `content-encoding` when rebuilding
the response.
2. Build failure under tsc: `import type { Context } from '@netlify/edge-functions'`
fails Next.js's TypeScript check because the package is only present in
the Netlify edge build environment, not in node_modules. Edge functions
run on Deno with Netlify-provided globals, not Node, so they're a
separate compilation unit. Add `netlify` to tsconfig.json's `exclude` so
`next build` (and the Cypress CI's prebuild) stops scanning the
directory. Netlify's edge function build still typechecks the files on
the server side with the correct types.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
viniciusgiles
approved these changes
Jun 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Migrate the docs site to
archive.sparkpost.comso the existingsupport.sparkpost.comhostname is free to be repointed at the new CloudFront redirect distribution (messagebird-dev/bird#2382, merged) that 301s every legacy URL to its closest bird.com counterpart. End goal: consolidate SEO link equity onto bird.com instead of fragmenting between two sites.This PR is half of a two-part change. The other half is the CloudFront distribution (
bird#2382). They have to land together for SEO — landing this PR alone (or far ahead of CloudFront) would deindexsupport.sparkpost.comwhile it still has all its inbound Google traffic, losing link equity that would otherwise transfer to bird.com. The edge function added here exists specifically to bridge that gap so the two pieces don't have to land simultaneously.Changes
1. Base URL change + noindex (for
archive.sparkpost.com)The new archive hostname needs to be configured everywhere and emit clear noindex signals so search engines drop it (and stop competing with bird.com).
next-sitemap.js: defaultsiteUrl→https://archive.sparkpost.com;robots.txtgeneration addsDisallow: /to the*policy.components/site/seo.tsx:<meta name="robots">content →noindex, nofollow(wasindex, follow, max-image-preview:large, ...).netlify.toml: addX-Robots-Tag: noindex, nofollowto the global[[headers]]block; CSPfont-srcsupport.sparkpost.com→archive.sparkpost.com(the prior value was hardcoded but functionally unused).2. Host-conditional noindex (preserves
support.sparkpost.comindexability during the gap)Without this, the changes above would apply equally to both hostnames the Netlify deploy serves — including the current production
support.sparkpost.com. That's incorrect during the period between this PR landing and the CloudFront cutover:support.sparkpost.comstill has all its Google traffic, and deindexing it before the 301s exist loses equity rather than transferring it.netlify/edge-functions/host-conditional-noindex.ts(new) — a Netlify Edge Function that runs on every request and, forHost: support.sparkpost.comonly:X-Robots-Tagfrom response headers<meta name="robots" content="noindex,...">from HTML responses/robots.txtwith a permissive version (User-agent: *, noDisallow)archive.sparkpost.comand all other hosts (deploy previews,*.netlify.app, etc.) pass through unmodified — noindex stays in place.After CloudFront cutover (out of scope here)
Once
support.sparkpost.comDNS flips to the CloudFront distribution, Netlify never sees those requests again — every viewer hits CloudFront and gets 301'd to bird.com. At that point the edge function is dead code on the Netlify side. Remove it (and the netlify.toml comment pointing at it) in a small follow-up PR.Test plan
After deploy preview is up:
curl -I https://archive.sparkpost.com/docs/→ response includesX-Robots-Tag: noindex, nofollowcurl -I https://support.sparkpost.com/docs/→ response does NOT includeX-Robots-Tagcurl -s https://archive.sparkpost.com/robots.txt→ containsDisallow: /curl -s https://support.sparkpost.com/robots.txt→ permissive (User-agent: *), noDisallowhttps://archive.sparkpost.com/docs/...→ contains<meta name="robots" content="noindex, nofollow">https://support.sparkpost.com/docs/...→ does NOT contain that meta tag🤖 Generated with Claude Code
Note
Medium Risk
SEO and crawl behavior depend on correct Host-header branching in the edge function; a bug could deindex support.sparkpost.com or leave the archive indexed.
Overview
Repoints the docs site’s default canonical URL to
archive.sparkpost.comand applies noindex everywhere the build emits SEO signals: robots meta inseo.tsx, globalX-Robots-Taginnetlify.toml,Disallow: /in generatedrobots.txtvianext-sitemap.js, plus CSPfont-srcupdated fromsupport.sparkpost.comtoarchive.sparkpost.com.Because one Netlify deploy still serves
support.sparkpost.comuntil CloudFront/DNS cutover, a new Netlify edge function (host-conditional-noindex.ts) runs only for that host: it serves a permissive/robots.txt, setsX-Robots-Tag: all, and strips the noindex robots<meta>from HTML so production support URLs stay crawlable and don’t lose equity before 301s to bird.com.archive.sparkpost.comand other hosts pass through with noindex unchanged.tsconfig.jsonexcludes thenetlify/folder so edge TypeScript isn’t typechecked with the Next app.Reviewed by Cursor Bugbot for commit 54d49ea. Bugbot is set up for automated code reviews on this repo. Configure here.