What Is a Robots.txt File, and Does a Small Business Website Need One?

By Bill Stilwell · Last reviewed October 16, 2026

The short answer

A robots.txt file is a small text file at the root of your website that tells search engine crawlers which parts of the site they may visit. Most small business sites need only a simple one that allows everything and lists the sitemap. Robots.txt does not keep a page out of Google. To keep a page out of search results, use a noindex tag or a password instead.

What this means

Search engines send programs called crawlers to read your pages. Google’s is called Googlebot. Before crawling, they check yoursite.com/robots.txt for rules.

A robots.txt rule says “don’t visit these addresses.” Google says the file is mainly used to keep crawlers from overloading your site with requests. It’s not a lock. Google says reputable crawlers follow it, but others may not.

Here is what a simple robots.txt for a small business might look like:

User-agent: *

Allow: /

Sitemap: https://www.example.com/sitemap.xml

The first line means “these rules are for all crawlers.” The second allows everything. The third tells crawlers where your sitemap is (see 19.5.8 XML Sitemaps).

Say Northside Auto Repair’s developer built the new site on a test address and blocked all crawlers with “Disallow: /” so the unfinished site wouldn’t show up. At launch, nobody removed that line. The new site quietly dropped out of search. It’s an easy mistake to make, and it’s why you check robots.txt on launch day.

Why it matters

One wrong line can hide your whole site. “Disallow: /” tells crawlers to stay out of everything.
Blocking isn’t hiding. Google says a blocked page can still show in results if other pages link to it, just without a description. If you need a page out of Google, use noindex (see 19.5.2 Index Control).
Blocking a page can stop noindex from working. If Google can’t crawl the page, it never sees the noindex tag.
Blocking style or script files can break how Google sees your page. Google says not to block those files if the page is hard to understand without them.

Who this applies to

Every business with a website. Most website builders create a robots.txt for you. You still need to check it, especially after a launch or rebuild. If your builder has a setting like “hide site from search engines,” that setting often controls robots.txt or a noindex tag. Google notes that on hosted builders you may need to use the builder’s search settings instead of editing the file.

How to do it

Open your browser and go to your domain followed by /robots.txt, for example www.example.com/robots.txt.
Read it. Look for “Disallow: /” on its own under “User-agent: *.” That blocks your whole site.
Check that your service, location, contact, and about pages aren’t blocked.
Make sure it lists your sitemap with a “Sitemap:” line and the full address.
If you need changes, use your website builder’s search settings, or have your developer edit the file. Google says the file must be named robots.txt, sit at the root of the site, and be saved as plain UTF-8 text. UTF-8 is a standard text format. Don’t make it in a word processor.
Don’t use robots.txt to hide private pages. Put private pages behind a password.
In Google Search Console, open the robots.txt report. It shows whether Google fetched your file and flags errors.
Check robots.txt again on launch day for any new or rebuilt site (see 13.8 Website Launch Checklist).
If you want to set rules for artificial intelligence (AI) crawlers, read 19.13.2 AI Crawler Access first.

What proof to keep

A copy or screenshot of your current robots.txt with the date
A screenshot of the Search Console robots.txt report showing no errors
Notes on any change you made and why

Keep these in your website records folder.

Common mistakes

Leaving a staging block in place at launch. Always check robots.txt the day the new site goes live.
Using robots.txt to remove pages from Google. It doesn’t remove them. Use noindex.
Blocking a page and also adding noindex. Google can’t see the noindex if it can’t crawl the page.
Blocking Cascading Style Sheets (CSS) or JavaScript folders. Google may not understand the page layout without them.
Putting the file in a subfolder. It only works at the root, like example.com/robots.txt.
Listing secret page addresses in robots.txt. Anyone can read the file. It points people right to what you wanted hidden.

When to get help from a pro

Your web developer or a search engine optimization (SEO) specialist: if your site disappeared from search after a launch, or you see robots.txt errors in Search Console. Questions to ask:

Is anything in robots.txt blocking pages we want in search?
Are any style or script files blocked that Google needs?

Bill’s note

[Bill: add a short story or lesson here. For example, a site launch where a leftover “Disallow” line hid the new site, and how long it took to recover.]

If you’d rather have this done for you, SiteSmarty checks robots.txt and search settings as part of every website launch.

How I did this for SiteSmarty

[Date I did this:]
[What I actually did, step by step: what SiteSmarty’s robots.txt says and how I checked it]
[What it cost and how long it took:]
[What surprised me or what I’d do differently:]
[Screenshot or photo to add, with private info blurred: SiteSmarty’s robots.txt in a browser and the Search Console robots.txt report]
[Result so far:]

Questions people ask

Can robots.txt keep a page out of Google? No. Google says a blocked page can still appear if others link to it. Use noindex or a password.

Where does the robots.txt file go? At the root of your site, like example.com/robots.txt. Google says each site can have only one.

How do I check if my site is blocking Google? Open yoursite.com/robots.txt in a browser, then check the robots.txt report in Google Search Console.

Should I block CSS and JavaScript files? Usually no. Google says not to block them if the page is harder to understand without them.

What if I blocked my whole site by mistake? Remove the block, check the robots.txt report, and request indexing of your key pages with the URL Inspection tool in Search Console.

Audit check

Question: Does robots.txt allow crawlers to reach all pages meant to be in search, and list the sitemap?
Pass looks like: A robots.txt at the site root that doesn’t block key pages or needed style and script files, includes a Sitemap line, and shows no errors in Search Console.
Fail looks like: “Disallow: /” left from staging, blocked service pages, blocked CSS or JavaScript, robots.txt used to hide private pages, or no file checks after launch.
Evidence to collect: A dated copy of robots.txt and the Search Console robots.txt report.

Where this fits

Do this first: 13.2.1 Website Hosting, 13.2.3 Staging and Production Sites

Do this next: 19.5.2 Index Control

Sources

Google Search Central, "Robots.txt introduction and guide" (checked October 16, 2026)
Google Search Central, "Create and submit a robots.txt file" (checked October 16, 2026)
Google Search Central, "Block search indexing with noindex" (checked October 16, 2026)
Google Search Central, "Build and submit a sitemap" (checked October 16, 2026)

This page explains general rules and Bill's own experience. It is not legal, tax, or insurance advice. Talk to a qualified professional about your situation.