Hack defences: protecting wp-config

I have a suspicion about the cause of yesterday’s hack. devlounge.net had a similar infiltration, but by a friendly attacker who explained his methods: he accessed the wp-config.php file while the site was rendering in plain text (which I’m assuming can happen through an apache error / misconfiguration). PHP files weren’t being processed correctly, so their code was visible to all. As wp-config contains the mysql username/password, the attacker could then log into myphpadmin, change the admin email address and request a password reset.

How to protect against this? Well, WordPress 2.8 supports moving the wp-config file one directory higher (via @egwor). For sites in the root, like mine, that’s above the publicly-accessible directories, so wp-config is completely out of harm’s way. I’ve just done this, and all is working fine.

If the WP install is in a subdirectory, the wp-config can be protected by adding this to the .htaccess (from devlounge.net):

# protect wpconfig.php
<files wp-config.php>
order allow,deny
deny from all
</files>

That makes it inaccessible even in plain text mode. Got to be worth doing.

Hacked

(Update: I now have an inkling as to how the hack happened, and how to prevent it recurring. See update post.)

The email said ‘Password Lost and Changed for user admin’. It was from my blog. I hadn’t touched my blog all day – all week, in fact – so this was pretty odd. I tried to log in. I couldn’t. Crap. I requested a password reset. The email never arrived. Double crap.

I logged into myphpadmin and checked the admin email address in the raw SQL. It wasn’t mine. I changed it back and requested a WordPress password reset. While waiting I downloaded the apache access logs, and a quick scan showed somebody was inside my blog and messing around with the WordPress plugins. Crappity cockity crap.

The email arrived, and I logged in. Nothing was immediately amiss. The plugins list was normal, so I figured they must be editing the theme. Again, nothing was obviously wrong, but I activated a backup theme anyway. But when I went to the front page: blank screen. So I re-activated the original, which immediately offered to update itself to the latest version. Did I want to do this, even though I would wipe out all customisations? I said yes.

I also had a full SQL backup from last weekend, so I wiped my database and restored that. Phew. I’d put out the fire. Or so I thought.

I wanted to understand what had happened, so it was back to the access log. A proper look showed a plugin *had* been uploaded, although it wasn’t a valid WordPress plugin so didn’t appear on the list. Nevertheless, the uploaded .php file had been manually run. A quick FTP check showed it was still there, and upon examination it contained a zipped file, which it was capable of extracting. Oh, cock. I deleted it. The access log then showed Mr Hacker had been messing around with…my backup theme. I examined the index.php and it had been replaced with the same malicious code. Which I had obligingly activated. So I deleted that too.

I carried on grepping the access logs, and Mr Hacker was still running stuff. He wasn’t in the admin interface any more, though he’d tried to reset the password again, but was running all sorts of files with querystring extract-to-these-directory arguments. There was some kak in the WordPress root, and when I went to kill it I spotted the www.paypal.com directory. Jesus. I deleted everything I could see, then downloaded the most recent access log: wongablog.co.uk/evil-paypal-link was already getting hits from all over the world. I tried the link in Chrome, just to check it wouldn’t work, and Google already (!) had it on a phishing list. The page itself was dead, happily.

At this point I finally remembered I could block Mr Hacker’s IP, so I did that. I tidied up everything I could find, and – currently – all seems well.

Except: how the hell did he change the admin email address? The logs show he accessed wp-config.php, then went straight to the wp-login.php and requested a password reset. My password wasn’t cracked, it was bypassed. Viewing wp-config really shouldn’t do anything. I can only think it’s a WordPress vulnerability, though I’m on the latest version.

I’m not sure of the wisest course of action. The admin account is gone1 and all the passwords secure…but they were before. I should probably wipe everything and install WordPress from scratch, just to be sure. Even so, this shouldn’t have happened, and I’m not sure how to stop it happening again. Damian’s pointed out this plugin, which is now activated, but I’m still nervous.

Hmph. I hate spammers. It’s great that WordPress thinks to email administrators whenever a password changes, but it’s still lucky I was sitting here. I’m actually on a borrowed laptop, and I’m incredibly glad I set up remote access to my home pc. Setting up FTP software would have been tricky otherwise. I think this counts as good redundancy planning. I shall, just for a moment, pretend I’m Batman.

  1. Damian warned me about this years ago, but I never quite got around to it []

*waves*

Sorry for the cliché, but I really am sorry for not updating much of late. This blog is six years old today, but the last six months have seen sporadic updates at best. Truth is, I’m barely treading water at the moment. I’m actually getting a bit worried about myself.

Recently it seems I don’t get excited about anything, and I don’t talk about the things I like because I’m embarrassed by them. I also don’t even seem to have any opinions – I feel like an idiot when people are talking about interesting things, because I don’t even know what I think, let alone have anything to add to the conversation. I increasingly leave places feeling dull, stupid, annoying, and not liking myself very much. I think I’m stuck in a loop of worrying way too much what other people think, and I can’t kick myself out of it.

This coincides with your basic I-don’t-know-what-I’m-doing-with-my-life crisis. I’m middling-ok with computers, but am crap at the people side. I’m middling-ok at taking photos, but am going into the third year of uni completely disillusioned with my course. I’m a slightly better writer, in the right circumstances, but am seemingly unable to commit to the hours of practice required to get better. I’ve no idea where I’m going or what I want to do. Blah.

Sorry to whine. Just wanted to explain the quiet. I’m trying – really – but at the moment I’m losing. I’m sure things will improve soon (and I am well aware that my life is cushy by most current or historical standards) and I’ll try to update here more often. I love my little blog, and I don’t want to see it fade. Hopefully this is just a phase, and soon this post will seem like just another bad day.

18 months of spam fighting

I could never understand why web hosting companies were so terrible. No matter who I signed up with, my blog was always unstable as hell, regularly becoming unresponsive for no apparent reason. I’d decided this was the Way of Things until I moved over to Damian‘s hosting, and he quickly found the cause: wongaBlog is regularly nuked by spammers. I’m on some zombie list – heaven only knows why – and the botnets regularly flood the site with such voluminous garbage that it falls over.

This was more than Damian had signed up for, but he valiantly fought off what amounted to DDoS attacks on his server. We tried plenty of remedies, and eventually had the blog locked down so tight we had to watch for regular users blocked in the crossfire. But the nature of botnets means you can never predict where the next wave will come from, and it was a continual battle to respond. Each new attack would take out the machine until Damian got there1, which was far from ideal. He did a great job keeping up with them, but eventually the spamming scale became too great and I was causing problems for other clients on the same server.

On his recommendation I’ve now moved over to Lunarpages, who use stupidly powerful servers even for the little guys. Things seem ok so far, though I expect an admin somewhere has had an interesting week.

But Damian deserves huge thanks for 18 months of swarm-fighting. It was way beyond the pay grade, and very kind of him – I’m most grateful. Hopefully his servers are more stable now.

  1. techy detail: such attacks would see the server load jump from 0.02 to 20 in seconds, taking out the slice and forcing a reboot []

Rest Stop

This blog will shortly disappear for a while, as I move to a new server in an attempt to deal with insane spam attacks. It’ll be offline for a few days minimum, as I may take the opportunity to redesign (and it’s not like I’ve had much to say recently anyway). See you soon.

Missing comments

Comments aren’t showing atm. Rather odd – they appear in ‘recent comments’ (oop, not any more), and each post knows how many there are, yet they stubbornly refuse to appear. Probably a database thing. Working on it…

Update: Sorted by Damian. Looks like my spammer chums corrupted the database, but it should be ok now.

Five years of blogging

My little blog is five years old today. Ahhhh. Please excuse a brief metabloggy interlude.

This site started when a friend of mine began blogging and I thought ‘that’s a good idea. I’ll steal it’. So I did, with a dubious design involving unicorns – really, I have no explanation for this – based around Movable Type 2.something. The name ‘wongaBlog’ came a few days later, based entirely on a short story I’d written in which the main character had a site called ‘wonkaBlog’, I think because he liked chocolate.

Since then it’s become a big part of my life, and I passed 3000 posts last month. It’s also, in no particular order::

  • briefly bathed in a pagerank of 5
  • been banned in libraries for swearing
  • pissed off two girlfriends – I think both eventually stopped reading entirely
  • been made fun of by the Guardian
  • regularly attacked by Russian spammers, which is quite the problem for my webhost but as close as I’m going to get to being in Spooks
  • served as a terribly cowardly way to ask someone out (not linking to this one, but it’s in there somewhere)
  • become the go-to-blog for Googlers of ‘wank-a-thon‘.

There’s lots more, but those are the ones that popped into my head while typing.

I love blogging. It gives me a chance to write, which is pretty much my favourite thing, and it’s also cathartic as hell. It helps me stay in touch with far-flung friends, as well as acting as a reasonably decent diary. And the very best aspect of blogging is a cliché, although none the less true for it: it’s the people you meet. I found myself in Bloggers4Labour a few years ago (not quite sure how, but I’m glad I did!), and I’ve met – both electronically and physically – lots of lovely people as a result. The same with various atheist / skeptical sites. It’s great, and makes me happy.

I don’t remember starting with a blogging goal, and I’ve never really developed one. Norm‘s Friday profilees are always asked for one piece of advice to a novice blogger, and a common reply is ‘know what you want to write, and who for’. I’ve never done that – this blog has always been for whatever I feel like at the time, with no plan or target audience – but I can see the attraction. For me the hardest part of blogging is discovering you’ve been read by people you respect, but don’t know personally. For a while it’s nigh on impossible not to second-guess yourself and think ‘oh god, what must they think of that‘ every time you click ‘post’. I can happily research and write a long piece on the problems of organised religion, say, then follow it up with a post on why I’m scared of women, or a snapshot of a particularly fascinating twig, or something. I’ve never come close to stopping, but it’s the thought of boring the hell out of interesting people that’s given me most pause.

I know this is silly. One of the best features of blogging is that it’s passive – if people don’t want to read, they don’t have to. It’s why the occasional hate-spewing trolls are so funny. But related, and trickier, is that many, if not most, people I know are by now aware of this site, and often mention it to me. Which is great, but does sometimes complicate things. I’ve a few posts permanently assigned to Drafts (over 100 at last count) because I never quite had the nerve to post, knowing they’d resonate with particular people. Sometimes it’s polite to spare people’s feelings when there’s no reason to post – I was at a really dull party this evening – but more often it’s over ideological disagreements. And it’s silly to worry about discussing those.

So my aim for the future is to worry less about what people will think. I shall write whatever I fancy, and if people stick around, great, if not, that’s fine too. And as long as I’m not rude and feel I can reasonably back up the more contentious stuff, that’ll do too – if I’m wrong, as I often am, people can tell me why.

Ok, enough wankery. I figure a five-year anniversary is an ok excuse, but I promise not to do this again for a long while. Incidentally, I was going to title this post after the appropriate anniversary material, so I hit Wikipedia for the list and discovered that four years is a ‘silk’ anniversary – how nice is that – while five years is ‘wood’. Wood. That’s crap. Plus I have no intention of ever typing ‘anniversary of wood’.

Finally, just to say thank you to anyone who comments, links to me from their own site, or just drops by and reads anything I write. It’s really very nice of you.

Overanalysis

I just discovered Overanalysis – a blog describing itself as “Dispatches from a personal educational journey on philosophy, religion, atheism and history”. Full of skepticism, atheism and general rationality, it’s definitely worth a look. Full disclosure: it’s (well) written by my Sydney-based cousin, who’s apparently another fan of The Skeptics’ Guide and therefore a classy chap. He’s currently relating his adventures infiltrating the Hillsong Church.

Spammers = bastards

Had something of a breakthrough today. Late this afternoon I noticed the website go down and called Damien, who in turn quickly spoke to the company that physically run the server. As the situation was ‘live’ they logged in to try to figure out what was happening. They immediately spotted 79 simultaneous connections to various pages on my site, all from porn IPs and zombie machines, that were completely overloading the system. Some very fast work with iptables calmed the situation somewhat, and we’re now blocking over 500 IPs.

Current theory: spam spikes have been causing the server to stop responding. They’re all trying to leave comments or trackbacks, and although Akismet / Bad Behaviour do a sterling job of blocking them, they still need to be processed. We had one at 1400, which would correspond with 0900 on the east coast of the US, and thousands of zombie machines getting turned on (possibly confirmation bias, but not an unreasonable possibility). All the recent tweaking of the server shored it up so it lasted longer, but there’s little the average server can do against what amounts to distributed denial-of-service attacks.

I didn’t know the extent of the spam problem. My statistics programs all require javascript to be enabled on the client, explaining why they only report ~300 visitors per day. Akismet outages told me that I was popular with spammers, but a) I assumed everyone was and b) I didn’t know about the spikes.

I don’t know whether this was the primary cause of the crashes or a contributing factor, but either way it’s a very helpful discovery. I’ve set up a duplicate site on another server to determine how well the site performs independent of evil spammers, but I’m hopeful this site should be much more stable now.