Sometimes our computers try a little too hard to help us and end up hurting us instead. In Norwegian we call it a “Bear’s Favor”. Such may be the case with your robots.txt file and Google’s upcoming Mobile-Friendly Ranking.
Test your site
If you haven’t done so already, head over to Google’s Mobile-Friendly Test page and enter your URL. If all is well and you’re running a responsive theme you should get a nice green all-clear sign meaning you won’t be penalized for mobile unfriendliness. However, there is a good chance your site will fail this test even if it is responsive because your robots.txt file is blocking your themes folder.
If so you’ll see something like the screenshot to the right.
If this is what you see your site will fail the mobile friendly test because your content is too wide, all your links are too close together, and everything is totally out of whack. Which is true.
Check your robots
The great thing is once you’ve run the test and failed, Google will provide you with directions to check what’s going on. What you need to do is go to Google Webmaster Tools and check your robots.txt file. Once on the page, enter the URL to your theme stylesheet (http://[yoursite.com]/wp-content/themes/[themename]/style.css) and see if it’s being blocked. If it is the tool will highlight the line in robots.txt that is blocking the stylesheet.
Fixing robots.txt is relatively easy. If you are not using an SEO plugins you may have to FTP or SSH into your site and change the file manually. If you are using Yoast’s WordPress SEO plugin you can edit robots.txt directly by going to Tools -> File Editor.
Easy as ?
13 replies on “robots.txt might be ruining your Mobile-Friendly Ranking with Google”
Very helpful article – thanks!
You should also check if any JS files are being blocked.
Had that problem recently on a photographers site which wasn’t indexed, until we found out that Google wasn’t able to render the site properly because /wp-includes/js/ (and therefore the core jQuery file) was blocked for bots.
It’s also possible to check the rendering of the page “as google sees it” right in Webmaster Tools which is often helpful in revealing such errors.
Great point Claudio.
Thanks, Morten…looks like excellent advice, and something I would never have thought of.
This is really strange.
If a client browser can load a site correctly this means it can correctly reach all css and js.
I don’t see why google bots shouldn’t be able to do the same.
In robots.txt you only “suggest” google to not index for SEO purposes some urls, you are not actually blocking anything.
Since the “Mobile test” purpose is emulating mobile browsers behaviour, I don’t see what this has to do with robots.txt
I had a client site with this mobile testing error, despite apparently having no robots.txt in the site root.
Turned out that the plugin “XML Sitemap & Google News Feeds” generates a virtual robots.txt, and also allows it to be customised from the WordPress dashboard. And some odd person had added the line “Disallow: */wp-content/”. Removing that line fixed it instantly.
????????????? ????? ??????? ???????????? ???????????? livescore ????? buaksib.com
I use your Simone theme. I did the Mobile Friendly Test, but still fail the test.
Content wider than screen
Links too close together
My website: http://www.pastelaccounting.co.za
Any help will be appreciated
It passes when I run the test.
Awesome! Thanks Morten, I have been racking my brains these past few days trying to figure out why my responsive WordPress theme is coming up as not mobile friendly during Google’s test. Turns out I was blocking wp-content in robots.txt as recommended some time ago.
Hello Morten, I am also facing the same issue i have a website http://www.gleamingmedia.com (it is a static site not in wordpress) which does not pass google mobile friendly test. Althogh it is responsive and works great on mobile. There is a robot.txt file in root which contains following text in it.
Any help will be appreciated
What I can see is that robots.txt block the mobile usage tester even though the page has been optimized. ¿Do I have to disable my robots.txt file until Google tests it again?
Howdy I am so excited I found your site, I really found you by
mistake, while I was searching on Askjeeve for something else, Nonetheless I am here now and would just like
to say kudos for a tremendous post and a all round thrilling blog (I also love the theme/design),
I don’t have time to read through it all at the minute
but I have bookmarked it and also added your RSS feeds, so when I
have time I will be back to read a great deal more,
Please do keep up the excellent work.
This is one of your best posts to date! I really appreciate that you added links to further reading to document and support your points.