Robots.txt for WordPress: How to Set Up WordPress Robots.txt

Following search engine optimization (SEO) best practices are paramount if you want your site to rank in search results and drive more qualified traffic to your pages. While there are dozens of items to check off your list, there’s one, in particular, you don’t want to forget — creating a robots.txt for WordPress. But what is a robots.txt file?

We’ve got the answer! On this page, we’ll answer questions like:

• What is a WordPress robots.txt file?
• Why do I need a robots.txt file for my WordPress site?
• How do I edit a robots.txt file with WordPress?
• How do I test my robots.txt?

Keep reading to find out more!

What is a WordPress robots.txt file?

WordPress robots.txt is a file that website owners create to tell search engines how to crawl and index their site. It provides search engines like Google with a guide for crawling and indexing to ensure that the most valuable pages get indexed to appear in search results.

Why do I need robots.txt for WordPress?

Robots.txt for WordPress is critical because it tells search engines which pages or folders they shouldn’t crawl. To help your site appear in search results, Google must crawl your pages and index them.

If you want these search bots to crawl your site effectively, you must guide them with your robots.txt file. The WordPress robots.txt doesn’t block crawlers from indexing certain pages on your site — that requires using a noindex meta tag. This file does, however, help guide crawlers away from those pages and focus on more important pages that need to be indexed, like your product pages, service pages, or content posts.

Using a robots.txt file for WordPress also helps you prevent these bots from slowing down your site. Without this file in place, bots can overload your site trying to crawl different pages, which can ultimately deliver a slow-loading experience. Considering slow-loading sites cost businesses $2.6 billion in revenue annually, you don’t want to risk it.

Overall, using a WordPress robots.txt file is beneficial to helping your business get your most essential pages ranking in search results without slowing down your site in the process.

How to edit robots.txt with WordPress

Now that you know why robots.txt is critical to your site appearing in search results, you’re ready to take action to help your site get crawled and indexed. There are two approaches to creating a robots.txt file. You can use a WordPress robots.txt plugin or opt to do it manually.

WordPress robots.txt plugin #1: All in One SEO

The first option for creating a robots.txt file is All in One SEO or AIOSEO.

This plugin comes with a robots.txt file generator that you can use to create and edit robots.txt files.     To edit your robots.txt using this plugin, follow these steps:

1. Log into your WordPress dashboard
2. Click on All in One SEO
3. Click Feature Manager
4. Scroll down to File Editor and click Activate
5. Go back to the All in One SEO menu
6. Click txt tab
7. Click Edit

From this point, you can input your WordPress robots.txt file and then save the changes for it to take effect on your site!

WordPress robots.txt plugin #2: Yoast SEO

Another WordPress robots.txt plugin you can use is Yoast SEO. Yoast SEO enables you to create and edit your robots.txt file on your website.     To use this plugin for editing your file, follow these steps:

1. Go to your WordPress Dashboard
2. Click on SEO
3. Click Tools
4. Click File Editor
5. Edit your robots.txt file

Once you edit your robots.txt file, you can add your text and save it for it to take action.

WordPress robots.txt. manual option

When you learn how to edit robots.txt with WordPress, you’ll find that you can opt to input your file manually instead of using a plugin. To do this, you need a file transfer protocol (FTP) like FileZilla to help you.

To create your WordPress robots.txt file this way, follow these steps:

1. Open a text editor (i.e., Notepad)
2. Type up the coding you want to use
3. Save the file as a txt file type
4. Connect your website to the FTP
5. Navigate to the public HTML folder
6. Upload the txt file from your computer to the server

How to test your WordPress robots.txt

Now that you know how to edit robots.txt with WordPress, you can add your specific rules and perimeters to crawl your site. Once you add your robots.txt file, you want to make sure that it’s working and that search engines can crawl your site. You can test your robots.txt for WordPress by using Google Search Console.

Google Search Console offers a robots.txt tester tool to help you see whether Google can crawl your site. Here’s how you can check:

1. Open the txt tester tool
2. Scroll through the robots.txt code to find the highlighted syntax warnings and logic errors
3. Type in the URL of a page on your site in the text box below
4. Select the user-agent you want to simulate in the dropdown list
5. Click the Test button

Once you hit the Test button, it will either say “ACCEPTED” or “BLOCKED.” This notification will tell you whether Google can crawl your page or not. Generally, you want to see “ACCEPTED” unless you’re testing the URL of a page you don’t want Google to crawl and index. If any URLs come up as “BLOCKED” that you don’t want to be blocked, you’ll have to go back into your robots.txt file, make changes, save them, and try testing the URL again.

Need help editing robots.txt for WordPress?

By adding a robots.txt for WordPress, you’ll help Google crawl and index your site more efficiently, so your most valuable pages can appear in search results.

But if you’re feeling confused or overwhelmed about how to add, edit, and test your WordPress robots.txt file, WebFX can help. We have a team of over 500 marketing experts that know how to optimize robots.txt files. Not only can we help you with that, but we can help you implement other SEO best practices, from keyword integration to content creation.

If you’re ready to get started, contact us online or call us today at 888-601-5359 to speak with a strategist about our WordPress SEO services!