First 10GB of photos moved
Tar, split, scp, cat, md5sum – the first 10GB of photos are now moved up to the new hosting server.
Slow business – but now it’s a job to re-knit the gallery2 database and see how that all performs.
Uploading mysql dumps to GoDaddy
Strictly a console guy – I’ve been struggling to get the big blog database dumps up to the new hosting. phpMyAdmin claims to support zipped dumps – but that doesn’t work. There are also timeouts in the console for the upload and import.
I finally fixed it by using scp to move the non-compressed dump to the hosting server; and then using the Hosting Control Center to restore the dump as if it was a backup.
It’s running right now – so hopefully I’ll have happy blogs again soon.
WordPress 3.1 beta 1
More testing of the server migration to the cloud – and also to note that WordPress 3.1 beta 1 is now live.
Some nice features – I’ll be kicking the tyres some more on this in the coming weeks.
Infrastructure – heavy lifting and planning
A trio of projects before the year-end – all interwined.
- Migration of mail from Google Apps to Hosted Exchange.
- Migration of DNS from current service provider to ‘someone new’
- Migration of blog/photos to ‘somewhere in the cloud’
Moving the mail isn’t that hard – it’s just making sure that mail doesn’t get dropped while the new MX and CNAMEs are propagating. The old mail will live on in Google Apps – the new stuff in hosted Exchange. The trickier part is making sure that ‘my customers’ get the right service – and can keep getting mail in Outlook or the web. Users eh.
Moving the DNS is part of the mid-term strategy to change ISP. Covad have been great to me since I moved to the US; sadly they are starting to show signs of decay. I need to support additional DNS records than the A, CNAME and MX records – no plans from Covad.
The final push is to move the blog servers out of the ‘home data centre’ and to a reliable, faster provider.
The ultimate aim is to divorce myself from Covad and the Static IP business DSL that has worked so well – and move to something that is much faster – but maybe without the SLA on the line itself.
xCache – PHP caching, performance and stability
I’ve been testing out xCache for a while – primarily as a PHP accelerator.
Early results were really promising – reducing page load times dramatically; and also reducing CPU load as common pages (i.e. the latest blog post and photos) were fed directly from the cache.
There seems to be some kind of memory leak/cache clean up issue with xCache 1.3 – I allocate some amount of RAM for cache (16MB, 64MB, 256MB – it really doesn’t matter) and at some point Apache/PHP starts eating up RAM, then starting to swap – and finally the server grinds to a halt.
xCache is off for now – I’ll keep investigating.
Overnight line problems
Any ideas?
Twice this week all connectivity has been lost – upstream of the CPE (on premise router).
The first was from 2100 to 0800:
The next from 2130 to 0430:
It looks like some kind of maintenance window from the Qwest who actually provision the line.
Uptime
The firewall/IDS/proxy box has been up for a year.
I’m happy with that.
Blog server implosion – fix and cause
I got an email on Saturday morning:
“I’m getting a message when I try and “post draft and edit online”. See pictures attached of the messages.”
Uh oh. Nothing had changed in the config of the web server for months – and adding extra disk space to the server wouldn’t cause this.
I looked at the Apache error logs – nothing. I couldn’t see anything that would be causing this. Typically it’s a permissions or xml-rpc problem that’s kicking up a complaint in Windows Live Writer.
Other blogs on the same server were working perfectly; I could upload via xml-rpc as well. Very strange.
Eventually I tracked down an alert in /var/log/warn that was flagging ‘cannot read inode bitmap’ – whenever I tried to upload an image via xml-rpc. Even stranger. This really didn’t make any sense – but it looked like early signs of a corrupt root filesystem and being unable to write to temp.
I dismounted everything and tried to fsck the disk – and then the world of pain unraveled. The entire root filesystem seemed to have junk – it’s ext3 so should be pretty robust. I’ve no idea what caused it – but the end result was that most of /etc was toasted and there were some 10,000 entries in lost+found.
The upside is that the mysql and web data are all on seperate disks – so really easy to reconstruct the server. I had backups of my PHP, mysql and Apache confs – as well as all the data. The only slog was updating the Apache/PHP/MySQL stack to the correct (current) versions for my uses.
What I learned:
- backups are great – but separating the data from the OS is a real winner
- backup the config files for the core apps
- document the correct versions of core apps. Currently Apache 2.2.10, PHP 5.3.2 and MySQL 5.1.3 – these all work together without problems
Total downtime – about eight hours. Real time spent fixing this – about three hours.
I also moved several of the blogs to WordPress 3.0 RC1 – it’s been really stable so far on the main blog. I also had to do a latin1 to utf8 conversion on one of the older blogs. Always painful – but a one time hit. I need to add that to the change control/validation for the next round of big updates.
Web server tweaking
Still trying to tweak the web server that little bit more.
More mysql optimisations – indexing, caching and some memory work.
I finally found a version of php5-eaccelerator that worked against PHP 5.2.12 on SLES 11. That seems to be working well right now.
LEGO club – Jan 2010
It’s been a good six months since the last LEGO robotics club at school – I should blog on what we did in that session.
This term it’s time to start up LEGO robotics again; we’ve limited the pre-school class to 4th and 5th grade – so we should have a pretty reasonable level of logic and construction skills.
I’m writing up the rules and the playbook for this session. We’re going to focus on three areas – similar plan to previous sessions:
– construction: gears, gear ratios and torque
– software: planning, prototyping, iterative troubleshooting
– project: communication, team work, documentation
The requirement is going to be:
Build a robot that can pull the largest mass on the sledge provided. A successful ‘pull’ will be over 50cm (20 inches)
Using the same robot chassis (you can change wheels and gears – but not rebuild the robot) cover a long, straight race course (~5m/~15 feet)in the shortest time.
Produce a display board for your project showing your design, thoughts, diagrams, photos and program.
Recent Comments