Please contact blacklistmgr@yahoo.com” – wtf?

I came into work today, and was met with some people telling me that our site had been blacklisted. It took a lot of runaround, and everyone denying involvement, but it sorta worked itself out. So what exactly happened?

Our Internet is hosted through XO Communications. Our domain is registered through Register.com, and hosted through Yahoo Small Business. We have an internal DNS that we run on a Windows 2003 SBS Server, and our Active Directory is “hitrunscore.local”.

First I called Yahoo support to confirm whether or not it was them. They said that any mention of an official email from Yahoo would include “@yahoo-inc.com” on the end of it. They said they’d look into it, but that it wasn’t anything this guy had heard of, nor could he duplicate it on his end.

Strange. I guess that means it’s localized between every PC in my company up to some point before Yahoo’s servers get involved.

So we do a Trace Route on the domain.

Comes up with the IP 68.142.205.137 – but it doesn’t deviate from the direct path between XO’s servers (algx, xo) to the backbone in my area (DTX-Dallas), to Yahoo’s servers. And it ends up on a yahoo.com server.

So I ping http://store.yahoo.com – which was also giving me the same issue…

That gives me the same address: 68.142.205.137. My initial inclination was that “OK, this is an ISP related thing. Maybe the DNS is bad for a record related to this IP address.” In reality, DNS handles name resolution – not IP resolution.

So, to make sure it wasn’t an in-house issue, I disconnected the WiFi Router, cleared the cache on the DNS server, Scavenged for old records, and restarted the service. That didn’t solve it, so I restarted the actual server. That too didn’t solve it.

Then I gave XO a call, and after I talked with 3 XO support groups, none of them gave me a real answer except that they didn’t know, and didn’t know anyone that could help.

The Unlikely Solution

While on the phone with them, I was trying to visit the IP again, and hit the F5 button several times on the page, out of frustration. And instead of getting the ‘please contact blacklistmgr@yahoo.com” ‘ message, I noticed that I intermittently got a legitimate webpage error.

This page actually had an error on it, and had some HTML source code. There was also a JavaScript in it being called from Yahoo’s official file servers. This changes things. It’s no longer and ISP or DNS or local issue. It’s a Yahoo issue.

Yahoo’s engineers are reportedly still looking into the issue, and it has happened once more since it was first fixed – but instead of lasting for an hour, it only lasted about 5 minutes.

From my chair, it appears to be a problem with the redirect from store.yahoo.com (68.142.205.137) to smallbusiness.yahoo.com (66.196.84.99). And because our domain was hosted on the yahoo stores platform, there is some kind of facade-ing going on with the domain in the address bar, vs what’s actually being called. This correlates with our search url http://search.store.yahoo.net/…/nsearch?catalog=hitrunscore&… where one search page is being used for multiple “catalogs”. It would not surprise me if one store domain is being used and subsequently masked for multiple catalogs in the same manner. Thus if store.yahoo.com couldn’t redirect properly, neither could the store.

I’m still baffled as to why it was a localized issue though, that only affected our building. And just as I typed this last part, the site went down again with the same issue. I’ll post an update when Yahoo has a real solution.

An Official Yahoo Update

I spoke with our Yahoo Small Business Representative, and he asked if there was anything that we had done this morning that could have caused a spike in business. The Yahoo systems look for and take appropriate actions to prevent Denial of Service attacks (DOS attack) on their systems. That was the case for us. While, yes, the redirect was broken, it was because we sent out an email campaign with the landing page and images hosted on our store. We sent that out to 40,000 people, all at the same time because of the increased open rate of doing it during the day. Unfortunately Yahoo’s servers detected it as a DOS attack on their system and blacklisted our IP. They have since Whitelisted our IP and recommended we stagger our email releases according to the timezone of the recipient.

So if you’re having the same issue, hopefully this helps pinpoint why you are, and you’ll know to take another look at what you’ve done recently that would have a caused a spike in your pageviews as Yahoo perceives them, and you can solve your issue sooner than we did.

Update…

After Christmas came around, it happened again, but we finally determined that it was Yahoo’s fault. They had made some security changes on their server to try and prevent DOS attacks and the like, but the settings were faulty and threw false positives. This blocked our IP again, and it went up and down several times during the day. They put our IP on a Whitelist, and to my knowledge it hasn’t happened in a while…if not since then. If this is happening to you, contact your Yahoo Store Representative and either ask him to have the engineers put your IP on a whitelist, or ask for the Top Tier Tech Support phone number and make the request yourself after they walk you through the troubleshooting stuff.