Any ideas?
Twice this week all connectivity has been lost – upstream of the CPE (on premise router).
The first was from 2100 to 0800:
The next from 2130 to 0430:
It looks like some kind of maintenance window from the Qwest who actually provision the line.
The firewall/IDS/proxy box has been up for a year.
I’m happy with that.
I got an email on Saturday morning:
“I’m getting a message when I try and “post draft and edit online”. See pictures attached of the messages.”
Uh oh. Nothing had changed in the config of the web server for months – and adding extra disk space to the server wouldn’t cause this.
I looked at the Apache error logs – nothing. I couldn’t see anything that would be causing this. Typically it’s a permissions or xml-rpc problem that’s kicking up a complaint in Windows Live Writer.
Other blogs on the same server were working perfectly; I could upload via xml-rpc as well. Very strange.
Eventually I tracked down an alert in /var/log/warn that was flagging ‘cannot read inode bitmap’ – whenever I tried to upload an image via xml-rpc. Even stranger. This really didn’t make any sense – but it looked like early signs of a corrupt root filesystem and being unable to write to temp.
I dismounted everything and tried to fsck the disk – and then the world of pain unraveled. The entire root filesystem seemed to have junk – it’s ext3 so should be pretty robust. I’ve no idea what caused it – but the end result was that most of /etc was toasted and there were some 10,000 entries in lost+found.
The upside is that the mysql and web data are all on seperate disks – so really easy to reconstruct the server. I had backups of my PHP, mysql and Apache confs – as well as all the data. The only slog was updating the Apache/PHP/MySQL stack to the correct (current) versions for my uses.
What I learned:
Total downtime – about eight hours. Real time spent fixing this – about three hours.
I also moved several of the blogs to WordPress 3.0 RC1 – it’s been really stable so far on the main blog. I also had to do a latin1 to utf8 conversion on one of the older blogs. Always painful – but a one time hit. I need to add that to the change control/validation for the next round of big updates.
Still trying to tweak the web server that little bit more.
More mysql optimisations – indexing, caching and some memory work.
I finally found a version of php5-eaccelerator that worked against PHP 5.2.12 on SLES 11. That seems to be working well right now.
I’ve been hand-hacking wp-includes/pluggable.php for several releases now. It just got old – so I decided to learn to write a real plugin to move the functionality of wp_redirect into my private plugin.
Here’s the issue. I have several sites that check that a user is logged in. These use runphp or exec-php so I can write/include PHP on the page:
<?php
/* Short and sweet */
global $user_level,$post,$user_login;
// get user information
get_currentuserinfo();
echo "Please wait … securing your connection …";
if ( $user_level == 0) {
// $user_level == 0 is anonymous or not logged in user
wp_redirect(get_option(‘siteurl’) . ‘/photos/sorry’);
}
else {
// $user_level >0 means they are logged in at least
wp_redirect(get_option(‘siteurl’) . ‘/wpg2’);
}
?>
The issue I’ve always had with this is that the standard wp_redirect writes the location information cleanly; because we are already in the page (and headers have already been written) Apache throws up and kills this:
[Thu Dec 31 04:51:18 2009] [error] [client 10.0.0.1] PHP Warning: Cannot modify header information – headers already sent by (output started at /www/foosite/wp-content/themes/regulus/header.php:5) in /www/foosite/wp-content/plugins/php-modify-headers-apache/php-modify-headers-apache.php on line 38, referer: http://foosite
The hand written fixes checked to see if headers had been sent; if so then do the naughty meta http-equiv refresh with the url instead.
if( !headers_sent() ) {
if ($is_IIS)
header("Refresh: 0;url=$location");
else
header("Location: $location");
} else
echo "<meta http-equiv='refresh' content='0;url=$location' />";
}
Testing the plugin now. Details later.
Upgrades to WordPress 2.9 on several of the production blogs – and it’s the same old issue with php header injection.
I’ve blogged about this before – and raised a trac ticket. I’m probably going to write a plugin to solve this one for good.
Here’s the change – around line 863 of wp-includes/pluggable.php
/* ** Remove header injection piece - fix for exec-php ** evilzenscientist - 27 Dec 09 ** originally from 28 May 08 ** ref http://trac.wordpress.org/ticket/2860
if ( $is_IIS ) {
header("Refresh: 0;url=$location");
} else {
if ( php_sapi_name() != 'cgi-fcgi' )
status_header($status); // This causes problems on I
header("Location: $location", true, $status);
}
}
endif;
** */
/** added new header injection and refresh ** http://trac.wordpress.org/ticket/2860 ** evilzenscientist - 28 May 2008 */ if( !headers_sent() ) { if ($is_IIS) header("Refresh: 0;url=$location"); else header("Location: $location"); } else echo "<meta http-equiv='refresh' content='0;url=$location' />"; } endif; /** end of change */
Recent Comments