Shopify Robots.txt: What It Actually Controls (and What It Doesn't)

Robots.txt is one of those files that gets more credit, and more blame, than it deserves. Merchants either ignore it completely or go in and start blocking things they shouldn't, usually because they read somewhere that "blocking pages helps SEO" without understanding what the file actually does.

Here's the short version: robots.txt tells crawlers where they're allowed to go. It doesn't remove anything from Google's index, it doesn't guarantee bots will listen, and used wrong, it can quietly strip pages out of search results that you actually wanted ranking.

Let me walk through what it does, what Shopify gives you out of the box, and where I'd actually recommend touching it.

What robots.txt is actually for

Your robots.txt file gives crawlers, Googlebot, Bingbot, and anything else that respects the standard, instructions on which parts of your site to crawl and which to leave alone.

The part people miss: it's a request, not a lock. Crawlers that follow the rules will follow it. Bots that don't care about the rules will ignore it entirely. And disallowing a URL in robots.txt doesn't remove it from the index, it just stops crawling. If Google already knows about that URL through a link somewhere, it can still show up in search results, just with no description, because Google was never allowed in to read it.

Where it earns its keep:

• Keeping crawl budget off pages that have zero SEO value
• Steering crawlers toward the pages that actually matter
• Keeping system and account pages out of the crawl queue (though noindex tags are the better tool for actually keeping something out of results)

What Shopify blocks by default

Shopify ships every store with a robots.txt file already configured, and it's genuinely decent out of the box. It blocks:

• /admin, your store's back end
• /cart, no SEO value, ever
• /orders and /account, customer-specific pages
• /checkouts, the checkout flow
• A handful of internal and system paths
• /collections/*+*, filtered collection URLs with multiple parameters stacked
• /blogs/*/tagged/*, blog tag archive pages
• /*/collections/*, nested collection paths

It also points crawlers at your sitemap automatically. For the overwhelming majority of stores, this default setup is fine as-is. It keeps the junk out and leaves products, collections, blog posts, and static pages open to be crawled and indexed.

When it's actually worth customizing

The defaults cover most stores, but a few situations genuinely call for changes:

You've got pages you don't want crawled. A staging collection, an old campaign landing page, anything that shouldn't be showing up in search.

Sort and filter parameters are creating duplicate content. URLs like ?sort_by=best-selling technically create a new URL for content that already exists elsewhere. Some stores choose to block these outright.

Vendor archive pages are indexing. Shopify auto-generates /collections/vendors/[vendor-name] pages. If you're not using these intentionally, they're often worth blocking.

You want tag pages crawlable. If you've built genuinely useful tag pages, real content hubs, not just a list of links, the default block is working against you.

You're running more than one sitemap. Custom sections sometimes need their own sitemap referenced.

How to actually edit it

Shopify has let stores customize robots.txt since 2021, through a template file called robots.txt.liquid.

To find or create it: go to Online Store, then Themes, then Edit Code (it's under the three-dot menu or Actions button depending on your theme editor version). Look in the Templates folder for robots.txt.liquid. If it's not there, you create it as a new template.

The file needs to start with Shopify's default rules, pulled in through Liquid's header tag for robots.txt, and then your custom Disallow or Allow lines go underneath that.

Customizations I'd actually make

Blocking one specific page. Add a Disallow line for that exact path. Simple.

Blocking filtered collection URLs. /collections/shoes?filter.v.size=10 and every variation of it can spiral into a duplicate content mess fast. Block the parameter patterns causing the damage.

Blocking your internal search results. /search?q=keyword generates thin, near-duplicate pages with no reason to be indexed. Block the path.

Adding a sitemap reference. If an app or custom build created a second sitemap, point crawlers to it here.

What not to touch

This is where merchants do real damage. A few things that should stay untouched no matter what:

CSS and JavaScript. Google renders your pages to understand them. Block the files that make rendering work, and you're actively hurting how your pages get evaluated. Shopify leaves these open by default, leave them that way.

Product images. Image search sends real traffic. Blocking your CDN or image paths takes that off the table entirely.

Collection pages. These are category-level assets that rank for broad, high-volume terms. Blocking them is giving away search real estate for no reason.

Every parameter URL, indiscriminately. Not all parameters cause duplicate content. Variant selectors, for instance, usually resolve to a proper canonical URL Google handles correctly on its own. Block the parameters actually causing problems, not all of them by default.

Using robots.txt to deindex a page. If a page needs to come out of the index, that's a noindex tag or a sitemap removal, not a robots.txt block. Block it in robots.txt and Google can't even see the noindex tag, so you end up with a page that stays indexed with no content behind it. That's the worst possible outcome.

Does crawl budget actually matter here?

Crawl budget is simply how many pages Googlebot is willing to crawl on your site in a given stretch of time. For most Shopify stores, anything under roughly 10,000 pages, this isn't something to lose sleep over. Google crawls small and mid-sized stores thoroughly regardless of what robots.txt says.

It starts to matter once you're dealing with:

• Very large catalogs (50,000+ product pages)
• Massive filtered URL combinations
• Parameter-driven URLs that create near-infinite crawl paths

At that scale, robots.txt becomes a real lever: pointing crawl budget at products and collections, and away from filtered views, tag archives, and search result pages that add nothing.

Checking your work after you make changes

Once you've edited robots.txt, verify it actually did what you intended:

Google Search Console. Crawl Stats (under Settings) shows how Googlebot is actually crawling your site. The URL Inspection tool lets you test individual URLs against your current robots.txt rules.

Third-party robots.txt testers. Search Console's built-in tester has been deprecated, but plenty of standalone tools do the same job, testing specific URLs against your rules.

A full site crawl. Run your store through Screaming Frog or Sitebulb. Both respect robots.txt by default, so anything they can't reach is a page Googlebot can't reach either. It's a fast way to catch accidental blocks.

Meta robots tags: the more precise tool

For controlling individual pages, meta robots tags do a more surgical job than robots.txt.

noindex, follow, keeps the page itself out of the index but still passes value through its links. Good for thin pages that link to more important ones.

noindex, nofollow, keeps the page out and stops link equity from flowing through it. Reserve this for pages that are genuinely internal.

On Shopify, you can apply these through your theme code with conditional Liquid logic, through an SEO app, or via custom metafields if your theme supports them.

The decision tree is pretty simple:

• → One page needs to come out of the index: meta robots noindex
• → An entire URL pattern needs to stay uncrawled: robots.txt disallow
• → A page shouldn't be indexed but its links should still get followed: noindex, follow (robots.txt would block the links too, which isn't what you want)

A few other Shopify SEO quirks worth knowing

Duplicate product paths. Every product lives at both /products/item and /collections/category/products/item. Shopify handles this correctly with a canonical tag pointing back to the /products/ version, leave it alone. Trying to block the collection-path version in robots.txt just interferes with a system that's already working.

Pagination. Paginated collection pages (?page=2, ?page=3) come with proper rel=next/prev signals built in, and Google's pagination handling has gotten good enough that these don't need special treatment. Leave them crawlable.

Tag pages. Blocked by default, and for most stores that's correct, they're thin, list-only pages. If you've actually built them into real content hubs with unique value, that's the one case where unblocking them makes sense.

Where to start

Open Online Store, then Themes, then Edit Code, and check whether robots.txt.liquid already exists in your templates. If it does, read through what's actually in there. If it doesn't, you're running on Shopify's defaults, which is a perfectly fine place to be for most stores.

Then head into Google Search Console and run the URL Inspection tool against five pages that matter: your homepage, a top-selling product, a key collection page, a recent blog post. Confirm each one is crawlable and indexable.