I just discovered Overanalysis - a blog describing itself as “Dispatches from a personal educational journey on philosophy, religion, atheism and history”. Full of skepticism, atheism and general rationality, it’s definitely worth a look. Full disclosure: it’s (well) written by my Sydney-based cousin, who’s apparently another fan of The Skeptics’ Guide and therefore a classy chap. He’s currently relating his adventures infiltrating the Hillsong Church.
Comments have gone a bit weird (thanks, Ed). Not sure what’s causing it atm, but they do work sometimes. Bit late now and I’m at uni all day tomorrow, but shall take a look asap. Might have broken the theme, which could do with updating with the latest code anyway. Thought things were a bit quiet.
Had something of a breakthrough today. Late this afternoon I noticed the website go down and called Damien, who in turn quickly spoke to the company that physically run the server. As the situation was ‘live’ they logged in to try to figure out what was happening. They immediately spotted 79 simultaneous connections to various pages on my site, all from porn IPs and zombie machines, that were completely overloading the system. Some very fast work with iptables calmed the situation somewhat, and we’re now blocking over 500 IPs.
Current theory: spam spikes have been causing the server to stop responding. They’re all trying to leave comments or trackbacks, and although Akismet / Bad Behaviour do a sterling job of blocking them, they still need to be processed. We had one at 1400, which would correspond with 0900 on the east coast of the US, and thousands of zombie machines getting turned on (possibly confirmation bias, but not an unreasonable possibility). All the recent tweaking of the server shored it up so it lasted longer, but there’s little the average server can do against what amounts to distributed denial-of-service attacks.
I didn’t know the extent of the spam problem. My statistics programs all require javascript to be enabled on the client, explaining why they only report ~300 visitors per day. Akismet outages told me that I was popular with spammers, but a) I assumed everyone was and b) I didn’t know about the spikes.
I don’t know whether this was the primary cause of the crashes or a contributing factor, but either way it’s a very helpful discovery. I’ve set up a duplicate site on another server to determine how well the site performs independent of evil spammers, but I’m hopeful this site should be much more stable now.
Damian = linux ninja, and has been working very hard trying to restore some semblance of sanity to this site. Some mysql tweaks, coupled with local caching, appear to have made a difference, but it’s a bit of a waiting game to see whether it goes over again. We’ll see…
It honestly never occurred to me that my site might be so badly-constructed as to cause this many problems. I’m being an enormous pain to my host, so am going to do some testing offline + different servers etc. to try and track down exactly what’s breaking.
Probably won’t be around for a bit, and if the site keeps crashing we’ll take it down entirely, but anybody interested can follow my twitter updates.
Right. I think I’m back. For good this time.
For the last year, if not more, I’ve been plagued by mysql dropouts. I tried disabling themes and plugins, but nothing made any difference so I assumed it was my old hosts - Textdrive. Unfortunately the problems recurred after moving to my new hosting with Damian, and he worked waaaaaaay harder than one would reasonably expect of one’s webhost to track down and fix the problem.
It turned out to be something in the Wordpress database itself. The mysql server would sporadically go mental with hits to the wp_options table, to the extent that it brought down the entire system. I’m not sure whether this was the result of some broken plugin hooks or just corruption (can you export/import corrupted tables?), but whatever the cause, rebuilding the database from scratch and re-importing posts/comments seems to have solved the problem.
Ra.
RSS feeds will probably go mental as a result of manually re-importing posts from the last week - sorry about that - and I think the database of ’subscribe to comments’ has gone away, but neither are anything I’m too worried about. I’m back on www.wongablog.co.uk rather than wongablog.co.uk due to redirects having fights, but that’s no big deal - I’ll have to get over my irrational hatred of the www, though 
‘Into the past’ is broken currently as I’m scared of pages that make massive database queries - hopefully it’ll be back at some point. Other than that, please let me know if anything seems awry…
Down to a basic theme as there’s something, somewhere in the depths of my site, breaking the database and affecting other sites on the same server. I feel rather bad about this. It doesn’t seem to be caused by any of the obvious plugin candidates, so I’m leaning towards it being some odd code in the theme. Hopefully. I’ll strip and revamp my main theme (actually I may just start again, given how messy it’s become over the past couple of years).
They do everything they’re meant to, except appear. Working on it. Thanks for the heads up, L.
Update: This a caching problem on individual machines. If your comment doesn’t appear, a force refresh (before submitting) on the comment page - Ctrl-F5 on IE and Ctrl-Shift-R on Firefox - should clear it up. Thanks, Damian!
Hellos! Did you miss me? I’m back up and running on a zippy UK server, which should mean the end of the intermittent mysql downtimes I’ve been plagued with for the last year.
Many thanks to the ever-excellent Damian for all his help with the move, including cheerful receipt of rather disjointed 0330 emails
Off to Oxford this evening for a talk on the elimination of human aging1, but I’ll be setting free a head full of blog in the very near future.
- by “Dr Aubrey de Grey”, of whose name I am extremely jealous [↩]
Lots to say about Strictly, university, mental Labour ministers who think there’s a moral case for marriage-based tax breaks, and probably more Strictly, but I’m in the middle of a move to Exciting New Server atm so am freezing things until it goes through. It’ll hopefully only be a couple of days, but here are some ideas if you have trouble coping:
- Get a parrot
- Verify how much crockery will fit in the washing machine
- Ponder the luck of Anton du Beck
- Start a joke-a-day website that’s actually funny
- Read a book called, I think, Tumbleweed, about a knight with some kind of dragon for a horse. Possibly. It had a knight in it, anyway. Was good.
Hope that helps. Back in a bit.
So should be house-trained and post-toddle. Far, far from maturity, however.
Is out. Amongst other things, it adds tagging support and update notifications. Anybody using Ultimate Tag Warrior can import tags to the new (much faster) system (and should disable UTW before upgrading), but expect themes to break. I’ve had to remove ‘related posts’, and the tags page is a bit sparse. I expect new plugins in the next few days will fix these issues.
The cut-down theme didn’t affect downtime much, so I’ve reverted to the original. It was looking tired so I tweaked it a little. I’m still not happy, but it’s better than before. The search box is now linked to Google, rather than an internal search. Google can search the entire page contents, comments and all, rather than the previous system which was limited to posts. It’s less fun for me as I can’t view stats on local searches, but that’s no big deal.
The lack of difference makes me suspect the problems are more to do with my hosts than my site. I’ve been using mon.itor.us to monitor system uptime. It works well for server failures, but can’t detect database problems. If my website responds with a ‘cannot connect to database’ error, mon.itor.us still registers this as a positive response. It’s not possible to monitor a mysql server directly with their software, but I found a blog with a clever idea: monitor the phpmyadmin login page - that’ll go down with the database. I think the database is going down frequently, and hopefully this will confirm my suspicions.
I’ve been a little disappointed in my hosts - Textdrive. I can appreciate that their job is to keep computers up and running rather than to help maintain individual sites, but none of my recent queries have even been answered, even an (you’d think) easy one regarding my general server load and whether I’m reaching the limits of shared hosting. Shame.
I’ve had reasonably high traffic levels recently, with the svchost and Derren Brown posts getting lots of attention. As a result my website keeps falling over for unknown reasons. ‘Reasonably high’ is only ~500 visits per day, which isn’t much by the standards of many blogs, and I definitely shouldn’t be having these kind of problems. I turned on a caching program which this afternoon broke, serving blank pages. Argh. I’ve downgraded to a theme that’s very easy on the server and shall monitor it for a few days. Early indications are it’s not making much difference, which is just weird.
Going to play around with the website code to fix comments etc.. Things will likely break.
1018: Broken broken broken. Good news is, I get the same problems with the most basic theme. Bad news is, that means there’s a data/server problem of some kind. I’ll leave it on this theme while testing…
1048: Seems to work if I disable every plugin in the world…time to re-enable one by one
1307: This makes no sense.
1314: Everything’s working now. I didn’t do anything other than dis-and-re-enable all the plugins. Have discovered a possible Wordpress bug, however.