Staging privacy: keeping staging out of search engines

Every environment except live is private to search engines by default. That covers staging, premium staging, multidev, and preview environments. You do not need to turn anything on, and you cannot turn it off by mistake from WordPress.

Live is different. Avaloi adds none of this to live, so your live site is indexed as usual. This works like Pantheon's test and multidev environments: only live is meant to be found.

What is blocked

On a staging or multidev environment:

• Search engines such as Google, Bing, Yahoo, DuckDuckGo, Baidu, Yandex, and Apple get a 403 Forbidden answer.
• SEO crawlers such as AhrefsBot, SemrushBot, MJ12bot, DotBot, BLEXBot, and PetalBot get a 403.
• AI crawlers such as GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, anthropic-ai, CCBot, PerplexityBot, Bytespider, Amazonbot, Diffbot, cohere-ai, and Meta-ExternalAgent get a 403.
• Archivers such as the Internet Archive get a 403.
• Every response carries the header X-Robots-Tag: noindex, nofollow, noarchive, nosnippet, noimageindex, notranslate, including error pages, images, and other static files.
• /robots.txt answers User-agent: * and Disallow: /. The web server sends it, so a plugin or a robots.txt file in the site cannot change it. Crawlers may always read it.

Link previews still work. Facebook, X (Twitter), Slack, LinkedIn, Discord, and WhatsApp can fetch the page to show a preview when you share the link with a client. Tools such as curl, Avaloi's own health checks, and Cloudflare's health checks are not blocked either.

How the layers work

Avaloi keeps the environment private in four layers, so one layer failing does not expose it.

1. The web server. The environment's Nginx sends the noindex header on every response, answers /robots.txt itself, and refuses known crawlers by their user agent. The environment name the server reads decides this, so a live environment never gets it.
2. WordPress. Avaloi sets Settings > Reading > Discourage search engines (the blog_public option) when the environment is created, and again after every clone, reset, and content push into it. Those jobs copy another database, and live's setting would come with it, so the job sets it again after the copy and says so in its progress log. The Avaloi MU plugin also forces it: blog_public always reads as off, every page gets a noindex, nofollow robots tag, the WordPress XML sitemaps are off, and WordPress sends no pings, pingbacks, or trackbacks. Every wp-admin screen shows the notice This is a staging site. Search engines are blocked.
3. Cloudflare. Avaloi never turns on edge page caching for a staging or multidev environment, even if you turn edge caching on in its Tools tab; the job keeps it off and says why. Pages therefore always come from the server, where layers 1 and 2 apply. This uses no Cloudflare firewall rule, so it works the same for any number of environments. See Bot protection for the rules you control yourself.
4. A name that is hard to guess. A new staging or multidev environment gets a temporary domain with a random ending, such as elric-staging-k3x9q.avaloi.com, so nobody can find it by adding -staging to your live name. Environments created before this change keep the name they have.

Avaloi never lists staging or multidev domains in sitemaps, on public pages, or on the status page, and only sends them to the people who work on the site.

Promoting to live

A promote copies code by default, and live keeps its own search setting. When you also push the database from staging to live, Avaloi reads live's Discourage search engines setting before the copy and puts it back afterwards, so live does not stay hidden because staging was.

Certificates and certificate transparency

Every certificate a public authority issues is written to public certificate transparency logs, and anyone can search them. The certificate your visitors see at Cloudflare is the platform's wildcard for *.avaloi.com, which names no single site. Two other certificates do name the environment:

• the Let's Encrypt certificate Avaloi orders for each temporary domain, which secures the hop from Cloudflare to your server;
• the certificate for a custom staging subdomain you add, such as staging.example.com.

Those names appear in certificate transparency logs. The noindex header, robots.txt, and crawler blocking still apply, but treat a staging name as findable, not secret.

What you can still do

• Share the link. Anyone with the address can open the environment in a browser. The random ending makes the address hard to guess, but it is not a password.
• Rename the temporary domain. You can rename it from the Domains tab. A name you pick yourself is easier to guess, so keep a random part in it.
• Check it. Open https://<your staging domain>/robots.txt, or look for the X-Robots-Tag header in your browser's developer tools.

Not yet

• A password for staging and multidev. An optional HTTP password that keeps people without it out of the environment is not built yet. Until it is, do not keep anything on a staging or multidev environment that must stay secret.
• Crawler blocking at the edge. The edge cache Worker can also send the noindex header, answer /robots.txt, and refuse crawlers at Cloudflare before a request reaches the server. It is built but not deployed yet, so today the server does this.
• Environments created before this change get the web server layer the next time Avaloi writes their web server settings, for example when you change the PHP version or a cache setting. The WordPress layer applies once the server runs the current Avaloi MU plugin.