robots.txt might be ruining your Mobile-Friendly Ranking with Google

Sometimes our computers try a little too hard to help us and end up hurting us instead. In Norwegian we call it a “Bear’s Favor”. Such may be the case with your robots.txt file and Google’s upcoming Mobile-Friendly Ranking.

Test your site

An unstyled WordPress page
Screenshot of mor10.com without styles applied. Decidedly mobile-unfriendly

If  you haven’t done so already, head over to Google’s Mobile-Friendly Test page and enter your URL. If all is well and you’re running a responsive theme you should get a nice green all-clear sign meaning you won’t be penalized for mobile unfriendliness. However, there is a good chance your site will fail this test even if it is responsive because your robots.txt file is blocking your themes folder.

If so you’ll see something like the screenshot to the right.

If this is what you see your site will fail the mobile friendly test because your content is too wide, all your links are too close together, and everything is totally out of whack. Which is true.

Check your robots

The great thing is once you’ve run the test and failed, Google will provide you with directions to check what’s going on. What you need to do is go to Google Webmaster Tools and check your robots.txt file. Once on the page, enter the URL to your theme stylesheet (http://[yoursite.com]/wp-content/themes/[themename]/style.css) and see if it’s being blocked. If it is the tool will highlight the line in robots.txt that is blocking the stylesheet.

Fix robots.txt

Fixing robots.txt is relatively easy. If you are not using an SEO plugins you may have to FTP or SSH into your site and change the file manually. If you are using Yoast’s WordPress SEO plugin you can edit robots.txt directly by going to Tools -> File Editor.

Remove the offending line of text (probably Disallow: /wp-content/themes/), save the file, and go back to Google’s Mobile-Friendly Test page. Run the test again, the new robots.txt file will kick in, and you’ll see your site displayed in the preview with your styles and JavaScripts applied.

Easy as ?

19 Comments

  1. Very helpful article – thanks!
    You should also check if any JS files are being blocked.

    Had that problem recently on a photographers site which wasn’t indexed, until we found out that Google wasn’t able to render the site properly because /wp-includes/js/ (and therefore the core jQuery file) was blocked for bots.

    It’s also possible to check the rendering of the page “as google sees it” right in Webmaster Tools which is often helpful in revealing such errors.

  2. This is really strange.

    If a client browser can load a site correctly this means it can correctly reach all css and js.
    I don’t see why google bots shouldn’t be able to do the same.

    In robots.txt you only “suggest” google to not index for SEO purposes some urls, you are not actually blocking anything.

    Since the “Mobile test” purpose is emulating mobile browsers behaviour, I don’t see what this has to do with robots.txt

  3. I had a client site with this mobile testing error, despite apparently having no robots.txt in the site root.

    Turned out that the plugin “XML Sitemap & Google News Feeds” generates a virtual robots.txt, and also allows it to be customised from the WordPress dashboard. And some odd person had added the line “Disallow: */wp-content/”. Removing that line fixed it instantly.

  4. Awesome! Thanks Morten, I have been racking my brains these past few days trying to figure out why my responsive WordPress theme is coming up as not mobile friendly during Google’s test. Turns out I was blocking wp-content in robots.txt as recommended some time ago.

    Thanks buddy!

  5. Hello Morten, I am also facing the same issue i have a website http://www.gleamingmedia.com (it is a static site not in wordpress) which does not pass google mobile friendly test. Althogh it is responsive and works great on mobile. There is a robot.txt file in root which contains following text in it.

    User-agent: *
    Disallow:

    ———————–
    Website: http://www.gleamingmedia.com

    Any help will be appreciated
    Thanks
    Shivam Gupta

  6. Howdy I am so excited I found your site, I really found you by
    mistake, while I was searching on Askjeeve for something else, Nonetheless I am here now and would just like
    to say kudos for a tremendous post and a all round thrilling blog (I also love the theme/design),
    I don’t have time to read through it all at the minute
    but I have bookmarked it and also added your RSS feeds, so when I
    have time I will be back to read a great deal more,
    Please do keep up the excellent work.

Leave a Reply

Your email address will not be published. Required fields are marked *