Sometime this week we will be adding about 10 new themes. Before we do that we want to clean out the old themes that are only in use on abandoned blogs. Most of our current themes are about 5 years old. There are lots of newer themes available that take advantage of newer features in WP. (ex. threaded comments)

This afternoon I added a plugin which will do a one time pass through the blogs and if they blog hasn’t been updated in the last 8 months, it will be switched to the default theme. The plugin does one blog at a time, so I’m going to give it over night to run through the blogs. Tomorrow we will disable the ones no longer in use.

Updated: I got a bit distracted with a couple chores around the house. The maintenance is done. Outage lasted about 2 seconds.

Sometime this afternoon there will be a very brief outage. I have to make some server configuration changes and restart the web server. Typically, that restart takes less than 5 seconds. If there is an issue restarting it may take me a minute or 2 to fix. The maximum outage will be less than 5 minutes.

Before I get into the brief explanation, I did a few updates while moving the site. If you’ve been logged in you probably noticed the admin bar at the top of the screen. If you want to turn that off on either the front of the site or in the admin area, you can do that under Users -> Your Profile. Visitors to your site don’t see that adminbar.

I also had to disable the live comment preview plugin. It was having an error. Andrea & I are going to go through the plugins list and look for ones that have/need updates. WordPress 3.1 has only been out for a week or so and for some plugins we may have to wait a week or two before they are updated.

Due to an issue that someone reported a month or two ago, I found that an upgrade 2-3 years ago had not been applied to all of the blogs. While moving I ran a script on all the blogs and applied the updates if they had not already been applied.

Now the explanation

We had several outages over the last couple months that were a result of issues at the data center where HSJ was until yesterday. We were also having occasional short outages about 15 minutes long in the middle of the night, usually on the weekends. Those had been becoming more frequent. It turns out that it was actually our host that was shutting the server down on grounds of excessive resource usage. My guess at why they were mostly happening in the middle of the night is that is when automated backups were being done or the server was having automatic updates performed. Both of those things do use extra CPU.

We spent most of the weekend trying to figure out why the server went down several times on Friday. On both Saturday and Sunday Andrea and I were still trying to figure out what was going on because we couldn’t find any obvious problem on the server. It turns out that by using up CPU digging around on the server we were inadvertently triggering the host shutting down the web server.

The problem created by the repeated shutdowns is that a good percentage of our server load comes from feed services like feedburner and search engine crawlers. Those automated services run on a schedule and over the weekend more and more of them queued up waiting for the server to be up. On Sunday night, the server would be up less than 10 minutes and then the resource monitor would shut it down for about 15 minutes. That’s the reason we had to leave hsj offline until we had it on a new server.

In hindsight, looking through a downloaded access log, I saw evidence of someone trying to hack into the server through the webserver. That is most likely the original cause of the brief outages on Friday.

In any event, the site is now on a dedicated server which probably has about 10 times the server capacity of the server we were on. Even though it wasn’t how I wanted to spend my weekend, I’m really glad to have hsj moved to a better and more stable platform.

We apologize for the 2 days of downtime. I will write up a brief version of what happened over the weekend later this evening. If you see this your DNS has switched to the new server.

Tonight I did some maintenance to address the issue some had with saving draft posts. I did take a backup before I performed the maintenance. If you have any issues with your site related to the posts please let us know.

Just as a quick update on the work (or lack there of). I was sick for about two weeks since moving the site over to this server. This week I’m getting back on track and doing some catch up. This weekend I’m moving two more sites to this server so that we can cancel our subscription to an older server so that we don’t have to pay for another month of service. Hopefully, I’ll have some time next week to work on some of the outstanding issues.

The anti-spam plugin needed a small update to work properly with WordPress 3.0. So, you’ve probably seen an increase in spam since the upgrade on the weekend. I just updated the plugin, so the volume of spam should go down significantly again.

I’ve updated the available themes list & turned off themes that were not in use or only in use by a low number of blogs. I didn’t physically remove any themes, I just took them off the list of themes that can be selected.

This week we will be working on removing themes that weren’t being used at all & adding some new ones. We also have to go through the plugins and give them a thorough check over to ensure they have no issues with the new version of WordPress.