What the file does
At yourdomain.com/robots.txt, crawlers look for rules before fetching pages. Rules say which areas a given crawler should not visit. Reputable search engines follow them; malicious bots ignore them. It is a request, not a lock, so never use it to hide private information.
Basic syntax
A group starts with User-agent, naming a crawler or an asterisk for all. Then Disallow lines list paths to avoid, and Allow lines carve out exceptions. A typical small business file might read: User-agent: *, then Disallow: /wp-admin/, then Allow: /wp-admin/admin-ajax.php, then a Sitemap line giving the full sitemap address.
Paths are case sensitive and match from the start. A line reading Disallow: / blocks the entire site, which is correct only for sites under construction.
Blocking is not the same as hiding
If a blocked page has links pointing to it, a search engine can still list the address without a description. To keep a page out of results, use a noindex tag instead, and let crawlers reach the page so they can see it. Blocking in robots.txt and adding noindex together backfires, because the crawler never reads the tag.
For truly private content, use password protection.
Mistakes that cause real damage
The classic disaster: a developer blocks the site during a redesign and the rule stays after launch. Another is blocking CSS and script folders, which stops search engines from rendering pages correctly. Blocking image folders can remove pictures from image search. Typos, wildcards and trailing characters also change meaning.
After any site relaunch, load your robots.txt in a browser and read it line by line.
Testing and safe editing
Search Console shows how Google reads your robots.txt and whether pages are blocked. The URL Inspection tool tells you if a specific address is blocked. WordPress sites may generate a virtual file, edited via an SEO plugin, while other platforms offer a settings area.
Keep a copy of the working version before you edit. Change one rule at a time and retest. If unsure, leaving the default is safer than clever tweaks.
A safe starting file for a small site
For many small business sites, the safest file is nearly empty. A WordPress site might disallow only the admin folder while allowing the one file that front-end features need, then list the sitemap address. A site built on a hosted builder usually generates the file for you and often cannot be changed.
Before adding any new rule, ask what problem it solves. If you cannot name a specific section that wastes crawler time or exposes clutter, leave it alone. Over-blocking causes more harm than a few extra crawled pages ever will.
If you manage several sites or subdomains, remember that each hostname has its own file. A correct file on the main site does not protect or control a booking subdomain that is run by another service.
Common questions
Do I need a robots.txt file?
Not strictly. Without one, crawlers can visit everything. It is useful for pointing at your sitemap and excluding admin areas.
Can robots.txt remove a page from Google?
No. Use a noindex tag or the removal tools, and keep the page crawlable so the tag is seen.
Does it stop AI crawlers or scrapers?
Only the ones that choose to follow it. Treat it as a polite request.
This guide is general information. Search engines change how they work and their guidelines; check Google's current documentation for your situation. Nobody can guarantee a ranking.