Any ideas?
Twice this week all connectivity has been lost – upstream of the CPE (on premise router).
The first was from 2100 to 0800:
The next from 2130 to 0430:
It looks like some kind of maintenance window from the Qwest who actually provision the line.
The firewall/IDS/proxy box has been up for a year.
I’m happy with that.
I got an email on Saturday morning:
“I’m getting a message when I try and “post draft and edit online”. See pictures attached of the messages.”
Uh oh. Nothing had changed in the config of the web server for months – and adding extra disk space to the server wouldn’t cause this.
I looked at the Apache error logs – nothing. I couldn’t see anything that would be causing this. Typically it’s a permissions or xml-rpc problem that’s kicking up a complaint in Windows Live Writer.
Other blogs on the same server were working perfectly; I could upload via xml-rpc as well. Very strange.
Eventually I tracked down an alert in /var/log/warn that was flagging ‘cannot read inode bitmap’ – whenever I tried to upload an image via xml-rpc. Even stranger. This really didn’t make any sense – but it looked like early signs of a corrupt root filesystem and being unable to write to temp.
I dismounted everything and tried to fsck the disk – and then the world of pain unraveled. The entire root filesystem seemed to have junk – it’s ext3 so should be pretty robust. I’ve no idea what caused it – but the end result was that most of /etc was toasted and there were some 10,000 entries in lost+found.
The upside is that the mysql and web data are all on seperate disks – so really easy to reconstruct the server. I had backups of my PHP, mysql and Apache confs – as well as all the data. The only slog was updating the Apache/PHP/MySQL stack to the correct (current) versions for my uses.
What I learned:
Total downtime – about eight hours. Real time spent fixing this – about three hours.
I also moved several of the blogs to WordPress 3.0 RC1 – it’s been really stable so far on the main blog. I also had to do a latin1 to utf8 conversion on one of the older blogs. Always painful – but a one time hit. I need to add that to the change control/validation for the next round of big updates.
Still trying to tweak the web server that little bit more.
More mysql optimisations – indexing, caching and some memory work.
I finally found a version of php5-eaccelerator that worked against PHP 5.2.12 on SLES 11. That seems to be working well right now.
It’s been a good six months since the last LEGO robotics club at school – I should blog on what we did in that session.
This term it’s time to start up LEGO robotics again; we’ve limited the pre-school class to 4th and 5th grade – so we should have a pretty reasonable level of logic and construction skills.
I’m writing up the rules and the playbook for this session. We’re going to focus on three areas – similar plan to previous sessions:
– construction: gears, gear ratios and torque
– software: planning, prototyping, iterative troubleshooting
– project: communication, team work, documentation
The requirement is going to be:
Build a robot that can pull the largest mass on the sledge provided. A successful ‘pull’ will be over 50cm (20 inches)
Using the same robot chassis (you can change wheels and gears – but not rebuild the robot) cover a long, straight race course (~5m/~15 feet)in the shortest time.
Produce a display board for your project showing your design, thoughts, diagrams, photos and program.
I’ve been hand-hacking wp-includes/pluggable.php for several releases now. It just got old – so I decided to learn to write a real plugin to move the functionality of wp_redirect into my private plugin.
Here’s the issue. I have several sites that check that a user is logged in. These use runphp or exec-php so I can write/include PHP on the page:
<?php
/* Short and sweet */
global $user_level,$post,$user_login;
// get user information
get_currentuserinfo();
echo "Please wait … securing your connection …";
if ( $user_level == 0) {
// $user_level == 0 is anonymous or not logged in user
wp_redirect(get_option(‘siteurl’) . ‘/photos/sorry’);
}
else {
// $user_level >0 means they are logged in at least
wp_redirect(get_option(‘siteurl’) . ‘/wpg2’);
}
?>
The issue I’ve always had with this is that the standard wp_redirect writes the location information cleanly; because we are already in the page (and headers have already been written) Apache throws up and kills this:
[Thu Dec 31 04:51:18 2009] [error] [client 10.0.0.1] PHP Warning: Cannot modify header information – headers already sent by (output started at /www/foosite/wp-content/themes/regulus/header.php:5) in /www/foosite/wp-content/plugins/php-modify-headers-apache/php-modify-headers-apache.php on line 38, referer: http://foosite
The hand written fixes checked to see if headers had been sent; if so then do the naughty meta http-equiv refresh with the url instead.
if( !headers_sent() ) {
if ($is_IIS)
header("Refresh: 0;url=$location");
else
header("Location: $location");
} else
echo "<meta http-equiv='refresh' content='0;url=$location' />";
}
Testing the plugin now. Details later.
Upgrades to WordPress 2.9 on several of the production blogs – and it’s the same old issue with php header injection.
I’ve blogged about this before – and raised a trac ticket. I’m probably going to write a plugin to solve this one for good.
Here’s the change – around line 863 of wp-includes/pluggable.php
/* ** Remove header injection piece - fix for exec-php ** evilzenscientist - 27 Dec 09 ** originally from 28 May 08 ** ref http://trac.wordpress.org/ticket/2860
if ( $is_IIS ) {
header("Refresh: 0;url=$location");
} else {
if ( php_sapi_name() != 'cgi-fcgi' )
status_header($status); // This causes problems on I
header("Location: $location", true, $status);
}
}
endif;
** */
/** added new header injection and refresh ** http://trac.wordpress.org/ticket/2860 ** evilzenscientist - 28 May 2008 */ if( !headers_sent() ) { if ($is_IIS) header("Refresh: 0;url=$location"); else header("Location: $location"); } else echo "<meta http-equiv='refresh' content='0;url=$location' />"; } endif; /** end of change */
WordPress 2.9 beta 1 hits the streets today.
Looks pretty nifty – and everything seems to work ok so far.
From Mark Jaquith:
http://wordpress.org/wordpress-2.9-beta-1.zip
Big features to test:
• Basic image editing (rotate, flip, resize, crop)
• Post/Page image thumbnails. Enable the admin UI by declaring support
in your theme: add_theme_support(‘post-thumbnails’);
• Trash, with undo functionality, for posts, pages, comments
• Comment Meta table and functions — like Custom Fields/postmeta but
for comments
• Easy media embeds, oEmbed — paste a URL on its own item and have it
turn into embed code
• register_theme_directory() which enables plugins to bundle their own
themes, without copying (BuddyPress, primary example)
• Combo upgrader — get notified of plugin updates in the WP core
upgrader, as well as being informed of crowd-sourced compatibility
information for the plugins.
It’s bug-fixing and polishing time! Our priorities should be, in this order:
1. Fixing regressions in old features/behaviors
2. Squashing bugs in the new features
3. Polish
Couple of changes to WordPress 2.9 coming along – including deprecating streams.php and gettext.php in the includes.
There’s a ticket on it here – http://core.trac.wordpress.org/ticket/10890
The easy fix is to comment out the lines where the modules are loaded:
WeatherIcon.php55 # Fixes a bug in l10n.php where some guy decided there was no reason 56 # to load files which are required for their l10n functions if no 57 # language is defined in WordPress. 58 #require_once(ABSPATH . 'wp-includes/streams.php'); 59 #require_once(ABSPATH . 'wp-includes/gettext.php');
There’s an annoying DOS for WordPress doing the rounds – it’s blockable at the edge, using .htaccess, using a plugin, hand patching or by upgrading to 2.8.5
Kudos to the team for getting on top of this.
Here’s the scoop off wp-hackers:
http://wordpress.org/wordpress-2.8.5-beta1.zip
2.8.5 will probably release sometime in the next 24 hours. Changes since 2.8.4:
http://core.trac.wordpress.org/log/branches/2.8?action=stop_on_copy&mode=stop_on_copy&rev=12075&stop_rev=11811&limit=999
Summary:
* Fix for trackback DOS
* Removal of permalink_structure eval
* Remove some create_function() calls
* Disallow unfiltered uploads by default, even for admins. Enable it
again with define(‘ALLOW_UNFILTERED_UPLOADS’, true); in wp-config.php
* Add extra escapes here and there for some backside coverage
* Retire two old importers
* A few small bug fixes
This is mostly a security hardening release. There’s nothing exciting
unless you are concerned about the trackback DOS bug. Anyone who wants
to DOS your blog can do it regardless, but the trackback DOS bug makes
it easier for people to be annoying.
Recent Comments