How to Set Up a robots.txt File

What this means

robots.txt is a small text file at yoursite.com/robots.txt. It tells search engine crawlers which parts of your site they may visit, and where your sitemap is. A site audit flags sites where no robots.txt file could be found.

Why it matters

Without one, search engines assume they can crawl everything, which is fine for many small sites. But a robots.txt file is the standard place to point them at your sitemap, and to keep them out of areas with no value, such as admin pages or internal search results.

It's also worth knowing what yours says. A single wrong line in robots.txt can block your whole site from Google.

A safe starting file

For most small business sites, this is enough:

User-agent: *
Disallow:

Sitemap: https://yoursite.com/sitemap.xml

An empty Disallow: means nothing is blocked. Replace the sitemap address with your own (the guide on XML sitemaps explains where to find it).

How to add or edit it

• WordPress: WordPress serves a basic robots.txt automatically. SEO plugins such as Yoast (Tools, then File editor) and Rank Math (General Settings, then Edit robots.txt) let you edit it.
• Shopify: Shopify makes one for you. You can change it by adding a robots.txt.liquid template to your theme, but the default is right for most shops.
• Wix: the SEO settings have a robots.txt editor.
• Squarespace: Squarespace makes one for you and doesn't allow editing.
• A hand-built site: create a plain text file called robots.txt and upload it to the main folder of your site.

Mistakes to avoid

• Disallow: / blocks the entire site. It's often left on from when a site was being built. If you see it, and you want to be in Google, remove it.
• Blocking style sheets or scripts can stop Google seeing your pages properly. Don't block theme or plugin folders unless you know why.
• Using robots.txt to hide a page from Google. It stops Google visiting the page, but the page can still appear in results if other sites link to it. To keep a page out of Google, use a noindex tag instead (see the guide on blocked pages).
• Treating it as security. robots.txt is public and polite crawlers follow it, but it doesn't stop anyone opening the pages listed.

Checking your fix

Open yoursite.com/robots.txt in your browser and read it. Google Search Console also has a robots.txt report under Settings that shows the version Google last fetched.

If you use Duality, robots.txt is checked across the whole site, so the next monthly audit confirms it.