Try Tuts+ Premium, Get Cash Back!
Secure Your WordPress Against User-Agents and Bots

Secure Your WordPress Against User-Agents and Bots

Tutorial Details
  • Program: WordPress
  • Difficulty: Intermediate
  • Estimated Completion Time: 10 Minutes
  • Files to Edit: .htaccess (Make a backup, just in case!)

Lately there have been a lot of WordPress sites compromised only due to the bots that roam the world wide web! There are a lot of plugins out there which can protect your WordPress baby by blocking these “roguish” bots!

In this article you will be learning an easy and useful method of adeptly configuring your .htaccess file to filter these bots which can infect your website and can eat up your server resources. So get your .htaccess file ready for editing!


Step 1 Preparing the Code

The code mainly consists of bot names. I have added the most famous bots in here that I can think of. If there is some bot missing, please mention it in the comments.

The code is pretty straightforward. Go ahead and copy the code below and paste it in your .htaccess file.

# Bot Blocker
<IfModule mod_setenvif.c>
  SetEnvIfNoCase User-Agent ^$ keep_out
  SetEnvIfNoCase User-Agent (pycurl|casper|cmsworldmap|diavol|dotbot) keep_out
  SetEnvIfNoCase User-Agent (flicky|ia_archiver|jakarta|kmccrew) keep_out
  SetEnvIfNoCase User-Agent (purebot|comodo|feedfinder|planetwork) keep_out
  <Limit GET POST PUT>
    Order Allow,Deny
    Allow from all
    Deny from env=keep_out
  </Limit>
</IfModule>

Step 2 Testing the Code

To see whether the code is doing its job, I using recommend this website Bots VS Browsers. This website is a good place to simulate these types of attacks. Once on their website all you have to do is select any bot from the code, which you just added to your .htaccess file, and use that as the user agent. Enter the URL of your site and hit enter. If you see a “403 Error” this means that the code is doing its job. If not the code must’ve gotten messed up while being copied into your .htaccess file, so try again.


Step 3 Adding More Bots

Now you are familiar with the code and how to test it, we can add more bots to the code. You must have noticed the repetition in the code, and by using the same logic, you can add a dozen more bots to be blocked by setting the same parameters. Cool huh!

  SetEnvIfNoCase User-Agent (i-IS-evilBOT) keep_out

As you can see in the code above, now I am blocking the “i-IS-evilBOT” (which I just made up). Other than that the name of the bot is not case sensitive and you can add it as per your liking. Go to the Bots VS Browsers page and this time enter the user agent which I just created, and voila, you’ll see that this user agent which was added to my .htaccess file is also blocked! You can add as many bots as you want to be blocked separated with a pipe character “|”


Conclusion

I said in the beginning that there are many plugins which can do the same thing and you can avoid this editing. But by manually editing the .htaccess file you can effectively block bad user-agents and bots with better efficiency and better site performance!

Note: Want to add some source code? Type <pre><code> before it and </code></pre> after it. Find out more
  • http://lazaac.com lazaac

    Is there any method to applied those code into wordpress self-hosted in nGinx-based server?

  • benjamin

    How do these Bots compromise WP sites? I don’t understand this yet. Why is this an issue now?

    • https://plus.google.com/107464022992103498975?hl=en Bilal Shaheen
      Author

      They can actually leave spam comments or will use up all the resources and can cause down-time. Casper bot can even delete all your wordpress file.

  • http://cansurmeli.com C@N

    Hi.

    After adding this code to my .htaccess file(I checked before saving, it was correct) I got a 500 Internal Server Error. So I had to remove the code.

    And ideas? Could it be that this code overlapped with a plugin?

    • https://plus.google.com/107464022992103498975?hl=en Bilal Shaheen
      Author

      Try using the code provided in resources file. Otherwise I cannot think of any error. Or a better way to troubleshoot this is contacting support of your webhost provider. Hope this helps :)

  • http://www.mutterz.com Tricia Smith

    Thanks for this – I can see it being very useful. Is there anywhere you can find out the detail of further bots that it would be wise to block?

  • http://www.gateway-studios.net/ Euclides

    Thank you for sharing this technique, I never used it before so I will give it a try a protect more all my websites.

    • http://www.facebook.com/people/Bilal-Shaheen/100002670543697 Bilal Shaheen
      Author

      Glad you liked it : )

  • Pingback: Proteja o WordPress contra bots malignos - AmigoGeek

  • Pingback: Secure Your WordPress Against User-Agents and Bots | Shadowtek | Hosting and Design Solutions

  • http://www.elimcmakin.com Eli McMakin

    If you want to make your site even more secure, check out the 5G firewall, which has the above code and more:

    http://perishablepress.com/5g-blacklist-2012/

    When used in conjunction with cloudflare, askimet, and better wp security, my site is pretty locked down.

    • w1sh

      This should be a big, bright red, link at the top of this article. Thanks so much for sharing.

    • http://www.themenzclub.net Mack Hankins

      Wow, that is awesome. I’m going to put this on a website after work.

  • http://www.elimcmakin.com Eli McMakin

    Also, make sure you don’t overwrite your existing .htaccess file. Just put code like this somewhere in the middle or at the bottom. Otherwise you can cause problems.

  • Pingback: Why Liver Problems Require a Look at Celiac Disease | Open Knowledge

  • http://rentcarjakartatour.com Rent Car Jakarta Tour

    Can I use this trick together with the plugin?

  • http://moneyctl.com Nandita B.

    Hi Bilal ! Can you please suggest the best Plugin( Plugins ), which will replace the manual editing work.

    Thank you.

    • http://www.facebook.com/people/Bilal-Shaheen/100002670543697 Bilal Shaheen
      Author

      I would not suggest a plugin because its not a good idea to edit your htaccess file with a plugin, because a plugin will force rewrite the file and its not recommended. See if you can get some help from a friend to set up your htaccess file for your blog.

  • http://freakify.com Ahmad Awais

    I had been using a little plugin for this, buh thankx for a nice quick post.Hate plugins a lot.

  • http://icosmin.com Cosmin

    Or simply switch to CloudFlare.

    Their free plan takes care of bots and the like => less spam, save bandwidth ;)
    I have a relatively small website that I switched to CloudFlare 3 days ago (Feb 09, 2012) and here are some stats so far:

    8,903 Page views
    -> 6,609 from regular traffic
    -> 28 from crawlers/bots
    -> 2,266 from threats

    1,137 Unique visitors
    7 Unique crawlers
    311 Unique threats

    Requests Saved
    20,533 requests saved by CloudFlare
    32,645 total requests

    Bandwidth Saved
    285.4 MB bandwidth saved by CloudFlare
    552.6 MB total bandwidth

    Totally recommend them! (I am not in any way associated with CloudFlare, other than the fact that I use their free service)

    • http://www.uberweb.com.au alex

      Cloudfare is a great service but since it is reverse proxy so it will potentially mess with your analytics and adsense.

    • http://www.elimcmakin.com Eli McMakin

      CloudFlare is awesome, but I don’t believe CloudFlare is a failsafe for protecting your blog. You still want to put this stuff in your .htaccess file.

    • http://icosmin.com Cosmin

      Absolutely, I was probably just too excited :)

      I believe that CloudFlare in conjunction with such htaccess directives is a very efficient way of protecting WordPress and reducing load on your server. My spam dropped to less than half since I started using it. So did the bandwidth usage. If I add Amazon S3 for the static files, it’s a complete package :)

      @alex – don’t know about Adsense, but Google Analytics works as usual.

  • Pingback: .htaccessファイルを使ってBot等から身を守る方法 | 備忘録

  • http://www.fareham.co Joe Jenkins

    I’m not sure why, but this doesn’t seem to be working for me, I still see the page when testing.

    However, it is the same when using the 5G link.

  • gustav

    didnt work for me…
    returned a 500 server error.
    pasted the code from the source file after the .htacces code that was already there, so dont know what happened… does it need something extra from my host server in order to work?? greetings from Colombia

  • Pingback: Как защитить WordPress сайт от User-агентов и ботов | Wordpresso

  • Pingback: 207 Articles on Effective WebDesign - Winter 2011-2012 - PSD to HTML Blog

  • Pingback: Hardening WordPress: 8 Steps to a Secure Website | AUSWEB Web Hosting BLOG

  • http://www.bijusubhash.com Biju Subhash

    A great list of bad bots get from here.
    http://www.theadminzone.com/forums/showthread.php?t=69706

  • Gustavo

    it didnt work, when i upload it in my server the page goes with a 500 error… i paste the code at the end of the .htaccess file… help – and sorry for my english

  • Pingback: Andre's Web Dev: WP Class Blog » A Few Security Measures

  • http://twitter.com/WebWarBegin WebWarBegin

    absolutely great thing to learn for a newbie, i also like to edit .htaccess instead of using plugins