So, the part of the /etc/httpd/httpd.conf Apache 2.2 configuration file that takes care of the bad bots on this server had to be changed to allow for PayPal traffic despite them using empty (or, more precise “-” ) User Agent string.
PayPal traffic usually comes from 220.127.116.11 network (main eBay/PayPal IPN range) or from 18.104.22.168 network – PayPal Sandbox. If you never test your PayPal IPN (not a very good idea), you can open only the 66.x.x.x range but I would recommend doing both just in case you need to run a quick test – I guarantee you that once everything works, you will eventually forget that this PayPal workaround is there and will be chasing the wrong issue come next time something is up with PayPal IPN.
So, here is the recommended part of the httpd.conf file that’s responsible for filtering out bad bots which now has the PayPal IPN workaround. Please note that the actual list of bad bots used in there is by no means complete – I keep adding to it. Also note that some of the very common User Agent strings like python or libwww may actually belong to some Web-based tools that you may consider legitimate (like, link checkers and such) – so excercise judgement in implementing this. Just a personal note: I’ve been disallowing those for several years now with no visible negative impact.
<Location "/"> SetEnvIf User-Agent "^$" badUA SetEnvIf User-Agent "^-$" badUA SetEnvIf User-Agent "^.*(proximic|SkimBot|Sosospider|daum\.net|Ruby|HTTrack|clshttp|archiver|loader|email|nikto|miner|python).*" badUA SetEnvIf User-Agent "^.*(IndyLibrary|JikeSpider|AhrefsBot|winhttp|urllib|libwww\-perl|curl|wget|harvest|scan|grab|extract).*" badUA SetEnvIf User-Agent "^.*(MeMoNewsBot|Sogou|proximic|MJ12bot|R6_CommentReader|YoudaoBot|Nutch|SeznamBot|GarlikCrawler).*" badUA SetEnvIf User-Agent "^.*(Jakarta|80legs|binlar|larbin|Zing-BottaBot|Firefox\/2\.0\.0\.6|spbot|PiplBot|Synapse|Raloco).*" badUA SetEnvIf Remote_Addr "66\.211\." goodUA # the workaround for the stupid paypal IPN no-agent requests - main eBay/PayPal range SetEnvIf Remote_Addr "173\.0\." goodUA # the workaround for the stupid paypal IPN no-agent requests - PayPal Sandbox range Order Deny,Allow Deny from env=badUA Allow from env=goodUA Satisfy All </Location>