Categories
WordPress

robots.txt might be ruining your Mobile-Friendly Ranking with Google

Sometimes our computers try a little too hard to help us and end up hurting us instead. In Norwegian we call it a “Bear’s Favor”. Such may be the case with your robots.txt file and Google’s upcoming Mobile-Friendly Ranking.

Test your site

An unstyled WordPress page
Screenshot of mor10.com without styles applied. Decidedly mobile-unfriendly

If  you haven’t done so already, head over to Google’s Mobile-Friendly Test page and enter your URL. If all is well and you’re running a responsive theme you should get a nice green all-clear sign meaning you won’t be penalized for mobile unfriendliness. However, there is a good chance your site will fail this test even if it is responsive because your robots.txt file is blocking your themes folder.

If so you’ll see something like the screenshot to the right.

If this is what you see your site will fail the mobile friendly test because your content is too wide, all your links are too close together, and everything is totally out of whack. Which is true.

Check your robots

The great thing is once you’ve run the test and failed, Google will provide you with directions to check what’s going on. What you need to do is go to Google Webmaster Tools and check your robots.txt file. Once on the page, enter the URL to your theme stylesheet (http://[yoursite.com]/wp-content/themes/[themename]/style.css) and see if it’s being blocked. If it is the tool will highlight the line in robots.txt that is blocking the stylesheet.

Fix robots.txt

Fixing robots.txt is relatively easy. If you are not using an SEO plugins you may have to FTP or SSH into your site and change the file manually. If you are using Yoast’s WordPress SEO plugin you can edit robots.txt directly by going to Tools -> File Editor.

Remove the offending line of text (probably Disallow: /wp-content/themes/), save the file, and go back to Google’s Mobile-Friendly Test page. Run the test again, the new robots.txt file will kick in, and you’ll see your site displayed in the preview with your styles and JavaScripts applied.

Easy as ?

By Morten Rand-Hendriksen

Morten Rand-Hendriksen is a staff author at LinkedIn Learning and lynda.com specializing in WordPress and web design and development and an instructor at Emily Carr University of Art and Design. He is a popular speaker and educator on all things design, web standards and open source. As the owner and Web Head at Pink & Yellow Media, a boutique style digital media company in Burnaby, BC, Canada, he has created WordPress-based web solutions for multi-national companies, political parties, banks, and small businesses and bloggers alike. He also contributes to the local WordPress community by organizing Meetups and WordCamps.

13 replies on “robots.txt might be ruining your Mobile-Friendly Ranking with Google”

Very helpful article – thanks!
You should also check if any JS files are being blocked.

Had that problem recently on a photographers site which wasn’t indexed, until we found out that Google wasn’t able to render the site properly because /wp-includes/js/ (and therefore the core jQuery file) was blocked for bots.

It’s also possible to check the rendering of the page “as google sees it” right in Webmaster Tools which is often helpful in revealing such errors.

Thanks, Morten…looks like excellent advice, and something I would never have thought of.

This is really strange.

If a client browser can load a site correctly this means it can correctly reach all css and js.
I don’t see why google bots shouldn’t be able to do the same.

In robots.txt you only “suggest” google to not index for SEO purposes some urls, you are not actually blocking anything.

Since the “Mobile test” purpose is emulating mobile browsers behaviour, I don’t see what this has to do with robots.txt

I had a client site with this mobile testing error, despite apparently having no robots.txt in the site root.

Turned out that the plugin “XML Sitemap & Google News Feeds” generates a virtual robots.txt, and also allows it to be customised from the WordPress dashboard. And some odd person had added the line “Disallow: */wp-content/”. Removing that line fixed it instantly.

Awesome! Thanks Morten, I have been racking my brains these past few days trying to figure out why my responsive WordPress theme is coming up as not mobile friendly during Google’s test. Turns out I was blocking wp-content in robots.txt as recommended some time ago.

Thanks buddy!

Hello Morten, I am also facing the same issue i have a website http://www.gleamingmedia.com (it is a static site not in wordpress) which does not pass google mobile friendly test. Althogh it is responsive and works great on mobile. There is a robot.txt file in root which contains following text in it.

User-agent: *
Disallow:

———————–
Website: http://www.gleamingmedia.com

Any help will be appreciated
Thanks
Shivam Gupta

Howdy I am so excited I found your site, I really found you by
mistake, while I was searching on Askjeeve for something else, Nonetheless I am here now and would just like
to say kudos for a tremendous post and a all round thrilling blog (I also love the theme/design),
I don’t have time to read through it all at the minute
but I have bookmarked it and also added your RSS feeds, so when I
have time I will be back to read a great deal more,
Please do keep up the excellent work.

Comments are closed.