The robots.txt file tells search engine crawlers which parts of your site they may visit. A good one is short. A bad one can remove your whole site from Google without any warning.
A safe robots.txt for most WordPress sites
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/
Sitemap: https://example.com/sitemap_index.xml- Disallow /wp-admin/ keeps crawlers out of the dashboard.
- Allow admin-ajax.php — many themes and plugins load content through it, and Google needs it to render pages.
- Disallow internal search stops thousands of thin search-result pages being crawled.
- Sitemap points crawlers to your sitemap. Yoast uses
sitemap_index.xml; Rank Math usessitemap_index.xmltoo; core WordPress useswp-sitemap.xml.
Create yours with the Robots.txt Generator — pick the WordPress preset, add your sitemap URL and download the file.
Mistakes that hurt SEO
| Mistake | Why it's a problem |
|---|---|
Disallow: / | Blocks the entire site. Often left on after development. |
Blocking /wp-content/ or /wp-includes/ | Google can't load your CSS and JavaScript, so pages look broken to it. |
| Using robots.txt to hide a page | A blocked page can still appear in results. Use a noindex tag instead. |
| "Discourage search engines" left ticked | Settings → Reading. This adds noindex to every page. |
| Wrong sitemap URL | Open it in a browser to be sure it loads. |
How to add or edit robots.txt in WordPress
- With an SEO plugin: Yoast (Tools → File editor) and Rank Math (General Settings → Edit robots.txt) let you edit it from the dashboard.
- By file: upload a
robots.txtfile to the root folder (public_html) with your host's File Manager or FTP. A real file overrides WordPress's virtual one.
Test it
Open yoursite.com/robots.txt in a browser to see the live file, then use the robots.txt report in Google Search Console to confirm Google can read it. Need a sitemap for a non-WordPress site? Use the XML Sitemap Generator.
🛠️ Try Quick Tools Free
All our image tools are 100% free — no upload, no account, no limits. Works in your browser.
Browse All Free Tools →