I actually made this little feature for separating WordPress client pingback and trackback comments. I’m positive that you can also used it for comparing real trackback from spammer.

Most shell bots used perl http package

Here come the stats

According to my made-in-france bot tracker, libwww-perl is the most abused clients for sending code injection and Jakarta Commons-HttpClient is largely used for sending trackback SPAM.

90% are trackbacks spam

Simple rules

User agent string can be easily spoof so the target victim wont know the actual client. If the client is not coming from a blogging software or CMS its 90% spam oriented. And if the client is using generic user agent software HTTP package likes libwww-perl, Larbin, Nutch, et cetera it should be ignore.

How to banned user agent with htaccess

Banning these two user agents reduces my daily trackback spams to bearable amount. I’m gladly say these two user agent is the cause of many bandwidth misery. Both client is now honored in my htaccess rules ↓

<IfModule mod_setenvif.c>
SetEnvIfNoCase User-Agent "^libwww-perl*" shell_bots=1
SetEnvIfNoCase User-Agent "^Jakarta*" shell_bots=1
</IfModule>

<FilesMatch "(.*)">
Order Allow,Deny
Allow from all
Deny from env=shell_bots
</FilesMatch>

• Xirq xirq/0.1-beta (xirq; http://www.xirq.com; xirq@xirq.com)
• WebSearchBench WebSearchBench WebCrawler V1.0 (Beta), Prof. Dr.-Ing. Christoph Lindemann, Universität Dortmund, cl@cs.uni-dortmund.de, http://websearchbench.cs.uni-dortmund.de/
• Yahoo Search Japan robot Y!J-BSC/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
• NimbleCrawler Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.7) NimbleCrawler 1.11 obeys UserAgent NimbleCrawler For problems contact: crawler_at_dataalchemy.com
• Fastbot fastbot crawler beta 2.0 (+http://www.fastbot.de)
• Gigabot Gigabot/2.0/gigablast.com/spider.html
• Jambot Jambot/0.1.1 (Jambot; http://www.jambot.com/blog; crawler@jambot.com)

• Netluchs Netluchs/0.8-dev ( ; http://www.netluchs.de/; ___don\’t___spam_me_@netluchs.de)
• NutchEC2Test NutchEC2Test/Nutch-0.9-dev (Testing Nutch on Amazon EC2.; http://lucene.apache.org/nutch/bot.html; ec2test at lucene.com)
• Bigsearch Bigsearch.ca/Nutch-0.9-dev (Bigsearch.ca Internet Spider; http://www.bigsearch.ca/; info@enhancededge.com)
• UKWizz UKWizz/Nutch-0.8.1 (UKWizz Nutch crawler; http://www.ukwizz.com/)
• Ilial/Nutch ilial/Nutch-0.9 (Ilial, Inc. is a Los Angeles based Internet startup company. For more information please visit http://www.ilial.com/crawler; http://www.ilial.com/crawler; crawl@ilial.com)
• Pmoz Mozilla/5.0 (compatible; pmoz.info ODP link checker; +http://pmoz.info/doc/botinfo.htm)
• Holmes holmes/3.11 (OnetSzukaj/5.0; +http://szukaj.onet.pl)
• Flatlandbot flatlandbot/flatlandbot (Flatland Industries Web Spider; http://www.flatlandindustries.com/flatlandbot.php; jason@flatlandindustries.com)
• IDBot Mozilla/5.0 (compatible; IDBot/1.0; +http://www.id-search.org/bot.html)
• Spam Bot Mozilla/2.0 (compatible; NEWT ActiveX; Win32)
• Greaterera Mozilla/5.0 (compatible; heritrix/1.7.0 +http://www.greaterera.com/)
• GEXTEST-00393 gsa-crawler (Enterprise; GEXTEST-00393; gsasymbiosys@gmail.com,xeonbox4@gmail.com)
• Pagebull Pagebull http://www.pagebull.com/
• RSS One Engine RSS One Engine/0.72 (+http://www.rss-one.com)
• Dodgebot dodgebot/experimental
• Bot bot/1.0 (bot; http://; bot@bot.bot)
• Bigsearch Bigsearch.ca/Nutch-1.0-dev (Bigsearch.ca Internet Spider; http://www.bigsearch.ca/; info@enhancededge.com)
• FindLinks findlinks/1.1.4-beta1 ( http://wortschatz.uni-leipzig.de/findlinks/)
• ConveraCrawler ConveraCrawler/0.9e ( http://www.authoritativeweb.com/crawl)
• Blaiz-Bee Blaiz-Bee/2.00.5622 ( http://www.blaiz.net)
• KIT_Fireball KIT_Fireball/2.0
• ICC-Crawler ICC-Crawler(Mozilla-compatible;http://kc.nict.go.jp/icc/crawl.html;icc-crawl-contact(at)ml(dot)nict(dot)go(dot)jp)
• Pubblisito info@pubblisito.com- (http://www.pubblisito.com) il Sud dei Motori di Ricerca
• SkreemRBot Mozilla/5.0 (compatible; SkreemRBot +http://skreemr.com)
• WebAlta Crawler WebAlta Crawler/1.3.33 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU)
• Pumpkin blogsearchbot-pumpkin-3
• Mail.Ru Mail.Ru/1.0
• Mammoth Mozilla/5.0 (+http://www.eurekster.com/mammoth) Mammoth/0.1
• Attentio Attentio/Nutch-0.9-dev (Attentio\’s beta blog crawler; www.attentio.com; info@attentio.com)
• GurujiBot GurujiBot/1.0 (+http://www.guruji.com/en/WebmasterFAQ.html)
• Gigabot Gigabot/3.0 (http://www.gigablast.com/spider.html)
• Jobs.de-Robot Mozilla/5.0 (compatible; jobs.de-Robot http://www.jobs.de; jobsde@jobscout24.de) ( newsexpress e-mail: newsexpress-l@neofonie.de http://www.neofonie.de/loesungen/search/robot.html )
• ArabyBot ArabyBot (compatible; Mozilla/5.0; GoogleBot; FAST Crawler 6.4; http://www.araby.com;)
• VWBOT VWBOT/Nutch-0.9-dev (VWBOT Nutch Crawler; http://vwbot.cs.uiuc.edu;+vwbot@cs.uiuc.edu
• IWAgent IWAgent/ 1.0 - www.brandprotect.com
• Sirketcebot Sirketcebot/v.01 (http://www.sirketce.com/bot.html)
• Spock Crawler Spock Crawler (http://www.spock.com/crawler)
• Flatlandbot great-plains-web-spider/flatlandbot (Flatland Industries Web Spider; http://www.flatlandindustries.com/flatlandbot.php; jason@flatlandindustries.com)
• Nebulla Nebullabot/2.2 (http://bot.nebulla.de)
• EasyDL EasyDL/3.04 http://keywen.com/Encyclopedia/Bot
• LapozzBot LapozzBot/1.4 (+http://robot.lapozz.hu)
• WWW.fi crawler www.fi crawler, contact crawler@www.fi
• Uni-koblenz http://www.uni-koblenz.de/~flocke/robot-info.txt
• NimbleCrawler Mozilla/5.0 (Windows;) NimbleCrawler 2.0.1 obeys UserAgent NimbleCrawler For problems contact: crawler@healthline.com
• YodaoBot Mozilla/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spider/; )
• DAUM RSS Robot ELI/20070402:2.0 (DAUM RSS Robot, Daum Communications Corp.; +http://ws.daum.net/aboutkr.html)
• DAUM Web Robot Mozilla/4.0 (compatible; MSIE enviable; DAUMOA/1.0.1; DAUM Web Robot; Daum Communications Corp., Korea; +http://ws.daum.net/aboutkr.html)
• Changedetection Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; http://www.changedetection.com/bot.html )
• ICC-Crawler ICC-Crawler(Mozilla-compatible; http://kc.nict.go.jp/icc/crawl.html; icc-crawl-contact(at)ml(dot)nict(dot)go(dot)jp)
• Semager Semager/1.1 (http://www.semager.de/blog/semager-bots/)
• Multicrawler multicrawler ( http://sw.deri.org/2006/04/multicrawler/robots.html)
• NetinfoBot NetinfoBot/1.0 (http://netinfo.bg/netinfobot.html)
• Envolkspider envolk/1.7 (+http://www.envolk.com/envolkspiderinfo.html)
• CazoodleBot CazoodleBot/CazoodleBot-0.1 (CazoodleBot Crawler; http://www.cazoodle.com/cazoodlebot; cazoodlebot@cazoodle.com)
• RutterBot RutterBot(+http://www.aktienbetreuer.de/bot.html)
• Worio bot Mozilla/5.0 (compatible; woriobot heritrix/1.10.0 +http://worio.com)
• Tags2dir tags2dir.com/0.8 (+http://tags2dir.com/directory/)
• Combine Combine/3 http://combine.it.lth.se/
• Lawinfo-crawler lawinfo-crawler/Nutch-0.9-dev (Crawler for lawinfo.com pages; http://www.lawinfo.com; webmaster@lawinfo.com)
• FuseBulb FuseBulb.Com
• Earthcom Mozilla/5.0 (compatible; EARTHCOM/2.2; +http://enter4u.eu)
• Askpeter_bot Mozilla/5.0 (compatible; askpeter_bot/3.2; +http://www.askpeter.info)
• LapozzBot LapozzBot/1.5 (+http://robot.lapozz.hu)
• FAST-WebCrawler FAST Enterprise Crawler/6.4.18 (crawler@fast.no)
• BuiltWith Mozilla/5.0 (compatible; BuiltWith/0.1; +http://builtwith.com/bot.html)
• Hiiglespider Hiiglespider/0.1, Hiigle.com, http://hiigle.com/spider
• Page-store Mozilla/5.0 (compatible; heritrix/1.12.1 +http://www.page-store.com)
• Metacarta Mozilla/5.0 (compatible; heritrix/1.5 +http://www.metacarta.com)
• Multicrawler multicrawler (+http://sw.deri.org/2006/04/multicrawler/robots.html)
• LibertyW LibertyW (+http://www.libertyw.eu)
• BlogRefsBot Mozilla/5.0 (compatible; BlogRefsBot/0.1; http://www.blogrefs.com/about/bloggers)
• Holmes holmes/3.11 (http://morfeo.centrum.cz/bot)
• DataparkSearch DataparkSearch/4.47 (+http://dataparksearch.org/bot)
• ImageWalker ImageWalker/2.0 (www.bdbrandprotect.com)
• SeznamBot SeznamBot/2.0-test (+http://fulltext.sblog.cz/)
• Entireweb Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)
• BrightCrawler BrightCrawler (http://www.brightcloud.com/brightcrawler.asp)
• BabalooSpider BabalooSpider/1.2 (BabalooSpider; http://www.babaloo.si; spider@babaloo.si)

February 2008 - Crawler List