How To Fix a Broken robots.txt File in WordPress

It is a file every website should have. WordPress creates one by default and serves it to website visitors. Companies like Google use bots (software) to scan the internet for sites automatically and to read the contents of those sites. It’s called crawling, and the data set they are building is called their index. A robots.txt file tells the bots how to crawl your site, and what they are allowed to add to their index. Most bots read this file and honor its rules, so your site should have one, and you need to configure correctly. If it isn’t, you could be blocking Google on accident. Then you won’t get ranked, and won’t get the visitors that search engines can send, all of them will go to your competitors.

To check yours, visit http://yourdomain.com/robots.txt replacing yourdomain.com with your actual website’s domain. See which one you have, compare to the two examples above. Some hosting companies will give you different robots.txt files based on their settings. If you see anything different here, that is probably what’s happening.

Lucky for all of us WordPress has made this easy. Log in to your dashboard, at https://yourdomain.com/wp-admin/ again replacing yourdomain.com with your actual domain. The menu on the left has a Settings entry, click on it. More menu items will appear below Settings. Now click on Reading. On the Settings->Reading screen there is a checkbox at the bottom that says “Discourage search…” make sure that is unchecked and hit save. You’re done, it’s that easy. Watch the gif below to see it visually.

When you finish, recheck your robots.txt and see that the change worked. If you would like someone to do this for you, reach out to us here at Local SEO Company, and we’d be happy to fix this for free.

For a more comprehensive blog post on robots.txt files, and how to create custom ones you should check out Kinsta’s excellent article: here.