Select Page

Infrastructure – heavy lifting and planning

A trio of projects before the year-end – all interwined.

  • Migration of mail from Google Apps to Hosted Exchange.
  • Migration of DNS from current service provider to ‘someone new’
  • Migration of blog/photos to ‘somewhere in the cloud’

Moving the mail isn’t that hard – it’s just making sure that mail doesn’t get dropped while the new MX and CNAMEs are propagating. The old mail will live on in Google Apps – the new stuff in hosted Exchange. The trickier part is making sure that ‘my customers’ get the right service – and can keep getting mail in Outlook or the web. Users eh.

Moving the DNS is part of the mid-term strategy to change ISP. Covad have been great to me since I moved to the US; sadly they are starting to show signs of decay. I need to support additional DNS records than the A, CNAME and MX records – no plans from Covad.

The final push is to move the blog servers out of the ‘home data centre’ and to a reliable, faster provider.

The ultimate aim is to divorce myself from Covad and the Static IP business DSL that has worked so well – and move to something that is much faster – but maybe without the SLA on the line itself.

xCache – PHP caching, performance and stability

I’ve been testing out xCache for a while – primarily as a PHP accelerator.

Early results were really promising – reducing page load times dramatically; and also reducing CPU load as common pages (i.e. the latest blog post and photos) were fed directly from the cache.

There seems to be some kind of memory leak/cache clean up issue with xCache 1.3 – I allocate some amount of RAM for cache (16MB, 64MB, 256MB – it really doesn’t matter) and at some point Apache/PHP starts eating up RAM, then starting to swap – and finally the server grinds to a halt.

xCache is off for now – I’ll keep investigating.

Overnight line problems

Any ideas?

Twice this week all connectivity has been lost – upstream of the CPE (on premise router).

The first was from 2100 to 0800:

line-outage-4-5-oct-10

The next from 2130 to 0430:

line-outage-6-7-oct10

It looks like some kind of maintenance window from the Qwest who actually provision the line.

Blog server implosion – fix and cause

I got an email on Saturday morning:

“I’m getting a message when I try and “post draft and edit online”.  See pictures attached of the messages.”

Blog error

Uh oh. Nothing had changed in the config of the web server for months – and adding extra disk space to the server wouldn’t cause this.

I looked at the Apache error logs – nothing. I couldn’t see anything that would be causing this. Typically it’s a permissions or xml-rpc problem that’s kicking up a complaint in Windows Live Writer.

Other blogs on the same server were working perfectly; I could upload via xml-rpc as well. Very strange.

Eventually I tracked down an alert in /var/log/warn that was flagging ‘cannot read inode bitmap’ – whenever I tried to upload an image via xml-rpc. Even stranger. This really didn’t make any sense – but it looked like early signs of a corrupt root filesystem and being unable to write to temp.

I dismounted everything and tried to fsck the disk – and then the world of pain unraveled. The entire root filesystem seemed to have junk – it’s ext3 so should be pretty robust. I’ve no idea what caused it – but the end result was that most of /etc was toasted and there were some 10,000 entries in lost+found.

The upside is that the mysql and web data are all on seperate disks – so really easy to reconstruct the server. I had backups of my PHP, mysql and Apache confs – as well as all the data. The only slog was updating the Apache/PHP/MySQL stack to the correct (current) versions for my uses.

What I learned:

  • backups are great – but separating the data from the OS is a real winner
  • backup the config files for the core apps
  • document the correct versions of core apps. Currently Apache 2.2.10, PHP 5.3.2 and MySQL 5.1.3 – these all work together without problems

Total downtime – about eight hours. Real time spent fixing this – about three hours.

I also moved several of the blogs to WordPress 3.0 RC1 – it’s been really stable so far on the main blog. I also had to do a latin1 to utf8 conversion on one of the older blogs. Always painful – but a one time hit. I need to add that to the change control/validation for the next round of big updates.

LEGO club – Jan 2010

It’s been a good six months since the last LEGO robotics club at school – I should blog on what we did in that session.

This term it’s time to start up LEGO robotics again; we’ve limited the pre-school class to 4th and 5th grade – so we should have a pretty reasonable level of logic and construction skills.

I’m writing up the rules and the playbook for this session. We’re going to focus on three areas – similar plan to previous sessions:

– construction: gears, gear ratios and torque

– software: planning, prototyping, iterative troubleshooting

– project: communication, team work, documentation

The requirement is going to be:

Build a robot that can pull the largest mass on the sledge provided. A successful ‘pull’ will be over 50cm (20 inches)

Using the same robot chassis (you can change wheels and gears – but not rebuild the robot) cover a long, straight race course (~5m/~15 feet)in the shortest time.

Produce a display board for your project showing your design, thoughts, diagrams, photos and program.

Writing my first WordPress plugin – fixing the late header injection

I’ve been hand-hacking wp-includes/pluggable.php for several releases now. It just got old – so I decided to learn to write a real plugin to move the functionality of wp_redirect into my private plugin.

Here’s the issue. I have several sites that check that a user is logged in. These use runphp or exec-php so I can write/include PHP on the page:

<?php
/* Short and sweet */
global $user_level,$post,$user_login;
// get user information
get_currentuserinfo();
echo "Please wait … securing your connection …";
if ( $user_level == 0) {
// $user_level == 0 is anonymous or not logged in user
wp_redirect(get_option(‘siteurl’) . ‘/photos/sorry’);
}
else {
// $user_level >0 means they are logged in at least
wp_redirect(get_option(‘siteurl’) . ‘/wpg2’);
}
?>

The issue I’ve always had with this is that the standard wp_redirect writes the location information cleanly; because we are already in the page (and headers have already been written) Apache throws up and kills this:

[Thu Dec 31 04:51:18 2009] [error] [client 10.0.0.1] PHP Warning:  Cannot modify header information – headers already sent by (output started at /www/foosite/wp-content/themes/regulus/header.php:5) in /www/foosite/wp-content/plugins/php-modify-headers-apache/php-modify-headers-apache.php on line 38, referer: http://foosite

The hand written fixes checked to see if headers had been sent; if so then do the naughty meta http-equiv refresh with the url instead.

if( !headers_sent() ) {
if ($is_IIS)
header("Refresh: 0;url=$location");
else
header("Location: $location");
} else
echo "<meta http-equiv='refresh' content='0;url=$location' />";
}

Testing the plugin now. Details later.

WordPress 2.9 – exec-php and header injection

Upgrades to WordPress 2.9 on several of the production blogs – and it’s the same old issue with php header injection.

I’ve blogged about this before – and raised a trac ticket. I’m probably going to write a plugin to solve this one for good.

Here’s the change – around line 863 of wp-includes/pluggable.php

/*
** Remove header injection piece - fix for exec-php
** evilzenscientist - 27 Dec 09
** originally from 28 May 08
** ref http://trac.wordpress.org/ticket/2860
        if ( $is_IIS ) {
                header("Refresh: 0;url=$location");
        } else {
                if ( php_sapi_name() != 'cgi-fcgi' )
                        status_header($status); // This causes problems on I
                header("Location: $location", true, $status);
        }
}
endif;
**
*/
/** added new header injection and refresh
** http://trac.wordpress.org/ticket/2860
** evilzenscientist - 28 May 2008
*/
if( !headers_sent() ) {
if ($is_IIS)
header("Refresh: 0;url=$location");
else
header("Location: $location");
} else
echo "<meta http-equiv='refresh' content='0;url=$location' />";
}
endif;
/** end of change */

WordPress 2.9 beta 1

WordPress 2.9 beta 1 hits the streets today.

Looks pretty nifty – and everything seems to work ok so far.

From Mark Jaquith:

http://wordpress.org/wordpress-2.9-beta-1.zip
Big features to test:
• Basic image editing (rotate, flip, resize, crop)
• Post/Page image thumbnails. Enable the admin UI by declaring support
in your theme: add_theme_support(‘post-thumbnails’);
• Trash, with undo functionality, for posts, pages, comments
• Comment Meta table and functions — like Custom Fields/postmeta but
for comments
• Easy media embeds, oEmbed — paste a URL on its own item and have it
turn into embed code
• register_theme_directory() which enables plugins to bundle their own
themes, without copying (BuddyPress, primary example)
• Combo upgrader — get notified of plugin updates in the WP core
upgrader, as well as being informed of crowd-sourced compatibility
information for the plugins.
It’s bug-fixing and polishing time! Our priorities should be, in this order:
1. Fixing regressions in old features/behaviors
2. Squashing bugs in the new features
3. Polish