PayPal Instant Payment Notification (IPN) sends no User Agent

Came across this issue today while troubleshooting problems with PayPal payments on a site. PayPal kept sending the “PayPal Instant Payment Notification Warning” messages and eventually sent a “PayPal Instant Payment Notification Disabled” one and obviosuly no transactions could be automatically handled. Looking at the server logs revealed that they were getting a 403 Forbidden response, and the reason for that was the User Agent filter setup in the Apache HTTP server. Many bad bots send no User Agent or, more like PayPal does, User Agent is a useless 1-character string “-”. Filtering these out is a very common precaution for keeping bad bots at bay. This is by no means an ultimate bad bot protection, but if the authors of the bot did not even care to fill in the User Agent string, it means that the bot is not only sloppy but also generally up to no good. It was therefore much surprise to me to find out that PayPal developers are euqally inconsiderate of the Web developers’ needs.

So, the part of the /etc/httpd/httpd.conf Apache 2.2 configuration file that takes care of the bad bots on this server had to be changed to allow for PayPal traffic despite them using empty (or, more precise “-” ) User Agent string.

PayPal traffic usually comes from 66.211.170.0 network (main eBay/PayPal IPN range) or from 173.0.82.0 network – PayPal Sandbox. If you never test your PayPal IPN (not a very good idea), you can open only the 66.x.x.x range but I would recommend doing both just in case you need to run a quick test – I guarantee you that once everything works, you will eventually forget that this PayPal workaround is there and will be chasing the wrong issue come next time something is up with PayPal IPN.

So, here is the recommended part of the httpd.conf file that’s responsible for filtering out bad bots which now has the PayPal IPN workaround. Please note that the actual list of bad bots used in there is by no means complete – I keep adding to it. Also note that some of the very common User Agent strings like python or libwww may actually belong to some Web-based tools that you may consider legitimate (like, link checkers and such) – so excercise judgement in implementing this. Just a personal note: I’ve been disallowing those for several years now with no visible negative impact.

<Location "/">
    SetEnvIf User-Agent "^$" badUA
    SetEnvIf User-Agent "^-$" badUA
    SetEnvIf User-Agent "^.*(proximic|SkimBot|Sosospider|daum\.net|Ruby|HTTrack|clshttp|archiver|loader|email|nikto|miner|python).*" badUA
    SetEnvIf User-Agent "^.*(IndyLibrary|JikeSpider|AhrefsBot|winhttp|urllib|libwww\-perl|curl|wget|harvest|scan|grab|extract).*" badUA
    SetEnvIf User-Agent "^.*(MeMoNewsBot|Sogou|proximic|MJ12bot|R6_CommentReader|YoudaoBot|Nutch|SeznamBot|GarlikCrawler).*" badUA
    SetEnvIf User-Agent "^.*(Jakarta|80legs|binlar|larbin|Zing-BottaBot|Firefox\/2\.0\.0\.6|spbot|PiplBot|Synapse|Raloco).*" badUA
    SetEnvIf Remote_Addr "66\.211\." goodUA # the workaround for the stupid paypal IPN no-agent requests - main eBay/PayPal range
    SetEnvIf Remote_Addr "173\.0\." goodUA # the workaround for the stupid paypal IPN no-agent requests - PayPal Sandbox range
    Order Deny,Allow
    Deny from env=badUA
    Allow from env=goodUA
    Satisfy All
</Location>