Hacked
(Update: I now have an inkling as to how the hack happened, and how to prevent it recurring. See update post.)
The email said 'Password Lost and Changed for user admin'. It was from my blog. I hadn't touched my blog all day - all week, in fact - so this was pretty odd. I tried to log in. I couldn't. Crap. I requested a password reset. The email never arrived. Double crap.
I logged into myphpadmin and checked the admin email address in the raw SQL. It wasn't mine. I changed it back and requested a WordPress password reset. While waiting I downloaded the apache access logs, and a quick scan showed somebody was inside my blog and messing around with the WordPress plugins. Crappity cockity crap.
The email arrived, and I logged in. Nothing was immediately amiss. The plugins list was normal, so I figured they must be editing the theme. Again, nothing was obviously wrong, but I activated a backup theme anyway. But when I went to the front page: blank screen. So I re-activated the original, which immediately offered to update itself to the latest version. Did I want to do this, even though I would wipe out all customisations? I said yes.
I also had a full SQL backup from last weekend, so I wiped my database and restored that. Phew. I'd put out the fire. Or so I thought.
I wanted to understand what had happened, so it was back to the access log. A proper look showed a plugin *had* been uploaded, although it wasn't a valid WordPress plugin so didn't appear on the list. Nevertheless, the uploaded .php file had been manually run. A quick FTP check showed it was still there, and upon examination it contained a zipped file, which it was capable of extracting. Oh, cock. I deleted it. The access log then showed Mr Hacker had been messing around with...my backup theme. I examined the index.php and it had been replaced with the same malicious code. Which I had obligingly activated. So I deleted that too.
I carried on grepping the access logs, and Mr Hacker was still running stuff. He wasn't in the admin interface any more, though he'd tried to reset the password again, but was running all sorts of files with querystring extract-to-these-directory arguments. There was some kak in the WordPress root, and when I went to kill it I spotted the www.paypal.com directory. Jesus. I deleted everything I could see, then downloaded the most recent access log: wongablog.co.uk/evil-paypal-link was already getting hits from all over the world. I tried the link in Chrome, just to check it wouldn't work, and Google already (!) had it on a phishing list. The page itself was dead, happily.
At this point I finally remembered I could block Mr Hacker's IP, so I did that. I tidied up everything I could find, and - currently - all seems well.
Except: how the hell did he change the admin email address? The logs show he accessed wp-config.php, then went straight to the wp-login.php and requested a password reset. My password wasn't cracked, it was bypassed. Viewing wp-config really shouldn't do anything. I can only think it's a WordPress vulnerability, though I'm on the latest version.
I'm not sure of the wisest course of action. The admin account is gone1 and all the passwords secure...but they were before. I should probably wipe everything and install WordPress from scratch, just to be sure. Even so, this shouldn't have happened, and I'm not sure how to stop it happening again. Damian's pointed out this plugin, which is now activated, but I'm still nervous.
Hmph. I hate spammers. It's great that WordPress thinks to email administrators whenever a password changes, but it's still lucky I was sitting here. I'm actually on a borrowed laptop, and I'm incredibly glad I set up remote access to my home pc. Setting up FTP software would have been tricky otherwise. I think this counts as good redundancy planning. I shall, just for a moment, pretend I'm Batman.
18 months of spam fighting
I could never understand why web hosting companies were so terrible. No matter who I signed up with, my blog was always unstable as hell, regularly becoming unresponsive for no apparent reason. I'd decided this was the Way of Things until I moved over to Damian's hosting, and he quickly found the cause: wongaBlog is regularly nuked by spammers. I'm on some zombie list - heaven only knows why - and the botnets regularly flood the site with such voluminous garbage that it falls over.
This was more than Damian had signed up for, but he valiantly fought off what amounted to DDoS attacks on his server. We tried plenty of remedies, and eventually had the blog locked down so tight we had to watch for regular users blocked in the crossfire. But the nature of botnets means you can never predict where the next wave will come from, and it was a continual battle to respond. Each new attack would take out the machine until Damian got there1, which was far from ideal. He did a great job keeping up with them, but eventually the spamming scale became too great and I was causing problems for other clients on the same server.
On his recommendation I've now moved over to Lunarpages, who use stupidly powerful servers even for the little guys. Things seem ok so far, though I expect an admin somewhere has had an interesting week.
But Damian deserves huge thanks for 18 months of swarm-fighting. It was way beyond the pay grade, and very kind of him - I'm most grateful. Hopefully his servers are more stable now.
In need of a new domain
I need to change domain name. wongablog.co.uk regularly suffers massive spam attacks, and the resulting load takes down my site and anything else on the server (sites somewhat more important than little personal blogs). Individual attacks can be dealt with fairly quickly, but every new batch of zombie machines require individual attention, and it's never going to stop.
The easiest way of dealing with this is to upgrade to a ridiculously powerful server with lots of bandwidth. This costs a lot, though, and isn't an option.
The other solution is to ditch wongablog.co.uk and set up some kind of bot-befuddling forwarding system so links still work. This is my host's recommendation, and certainly seems sensible. So I need to set up a new domain in the not-too-distant future.
wordofwonga.co.uk and worldofwonga.co.uk have both been suggested, or I could just go back to the old wandwaver.co.uk. I've a couple of ideas, but nothing that particularly grabs me. Obviously I'm not asking people to put any effort into this, but should anything happen to occur, please let me know...thanks muchly 
Worth setting up the list, then
From a uni email telling me about the chaplaincy:
We have found that it is more inclusive to send send bulk messages rather than to our opt-in list so please delete this message if it is of no interest. Thank you.
Well, yes. That's what 'inclusive' means. The whole sentence works better if you say it like the aliens from Galaxy Quest.
Spammers = bastards
Had something of a breakthrough today. Late this afternoon I noticed the website go down and called Damien, who in turn quickly spoke to the company that physically run the server. As the situation was 'live' they logged in to try to figure out what was happening. They immediately spotted 79 simultaneous connections to various pages on my site, all from porn IPs and zombie machines, that were completely overloading the system. Some very fast work with iptables calmed the situation somewhat, and we're now blocking over 500 IPs.
Current theory: spam spikes have been causing the server to stop responding. They're all trying to leave comments or trackbacks, and although Akismet / Bad Behaviour do a sterling job of blocking them, they still need to be processed. We had one at 1400, which would correspond with 0900 on the east coast of the US, and thousands of zombie machines getting turned on (possibly confirmation bias, but not an unreasonable possibility). All the recent tweaking of the server shored it up so it lasted longer, but there's little the average server can do against what amounts to distributed denial-of-service attacks.
I didn't know the extent of the spam problem. My statistics programs all require javascript to be enabled on the client, explaining why they only report ~300 visitors per day. Akismet outages told me that I was popular with spammers, but a) I assumed everyone was and b) I didn't know about the spikes.
I don't know whether this was the primary cause of the crashes or a contributing factor, but either way it's a very helpful discovery. I've set up a duplicate site on another server to determine how well the site performs independent of evil spammers, but I'm hopeful this site should be much more stable now.
Oh Gmail, I hoped this day would never come
Gmail has officially jumped the shark. They're putting spam in the toolbar now!
This joke brought to you by my 13 year old self. The modern me is elsewhere.
Freeserve/Wanadoo/Orange email servers blacklisted in spam databases
Feb '07 Update: I've posted details of a way around this without changing email address. It's a little fiddly, but should work...
---
I've been trying to trace a problem with emails not reaching one of my clients. Emails from the same sender would sometimes get through fine, but often bounce back after a few days. I eventually managed to get a look at one of the bounced emails, which was very helpful.
It turns out that Freeserve have managed to get some of their email servers blacklisted in spam databases. Any ISPs that operate even basic spam filtering check all incoming email against the SpamCop or SORBS databases, to see whether the originating server is registered as a spammer. If so, the email is rejected and a few days later the sender will receive an undelivered mail report.
Fyi, Freeserve = Wanadoo = Orange, but I'm going to use Freeserve because I keep mistyping 'wanadoo'...
I did a little digging around, and unsurprisingly many people are having problems with this. It's flared up in recent days, but there are reports of problems with the servers - I looked up 193.252.22.157 - as far back as January of this year. This isn't the fault of the spam databases, nor is it difficult to get yourself removed from them. Freeserve seem to be entirely to blame.
This is a particularly annoying problem, as it must affect anybody with a Freeserve / Wanadoo / Orange account. It's unreasonable to ask every ISP/email host in the world to make an exception / disable their spam filters because of this, although that's what I've had to do in my client's case.
According to the SpamCop information page, the Freeserve servers are commiting the cardinal sin of bouncing back external emails, which afaik is quite an easy thing to fix! Freeserve presumably have a number of email servers of which only some are blacklisted, explaining why emails sometimes get through and sometimes don't. There doesn't seem to be any way of specifying which server you want to use, however.
I always used to like Freeserve, but given the long-running nature and severity of this problem, I can only recommend that nobody create a new Freeserve/Wanadoo/Orange account, and anybody on their system move to a different ISP.
Bombardment
Akismet is the WordPress blogging system's built-in comment spam filter. It's always worked pretty well for me, although my spam 'traffic' has never been very high. Having been away for five days, though, I came back to this:
Yeesh. I'd hate to have to sort that lot out manually.
Hello!
Hey, thanks for coming! I'm Andrew, and this is where my headthinks come to breed. They often meander around humanism and photography, but they're flighty and you can't trust them, so anything is possible.
Do feel free to email me - I get lonely:
Your hair looks swish, by the way - have you done something new?
Recent Comments
- vicky: going up this weekend having done all the welsh peaks except 4 snowdon, cant wait
on Climbing Snowdon. - vicky: what, how can u live in the 21 c and say that, probably a sexist thing
on Climbing Snowdon. - vicky: da iawn laura, cant believe what im hearing, all this fuss, anyway love the site, and i agree, normal people who swear and curse clomb mountains cos we live in the...
on Climbing Snowdon. - Andrew: Did a few quick test shots to check the light levels, then balanced the camera on the arm of a bench, set it for a 20s exposure and hoped for the best
on ISS. - lynsey: Wow, how did you take that photograph?
on ISS.
Don’t worry be happy
Spam comment caught by Akismet:
I think I'll go around blogs leaving this comment for real.