Infrastructure changes – Covad and GoDaddy frustrations

by ezs | Jan 3, 2011 | evilzenscientist, Uncategorized

Most of the way through the infrastructure changes at the moment.

A recap:

• Migration of mail from Google Apps to Hosted Exchange.
• Migration of DNS from current service provider to ‘someone new’
• Migration of blog/photos to ‘somewhere in the cloud’

Step one – the mail switch was relatively painless – it needed some careful planning – but zero downtime.

• signup for Hosted Exchange (actually the Microsoft Exchange Labs Friends and Family program)
• DNS changes (mainly CNAME work to prove ownership)
• family education (the hard part)
• DNS changes (MX records and webmail A and CNAME records)
• reconfiguration of email clients (Outlook, phones, devices etc)

As I wrote a couple of months ago – the old mail lives on at Google Apps. Everything new is in Exchange.

Step two of the move was more complex and painful. I decided to change the DNS hosting with a consolidation of the various registrars that I’d used over the past decade. What should have been a week-long process of sign-up, DNS unlock, auth code request and move – took most of my time.

I moved from register.com and Network Solutions (resold by Covad). Getting the domains unlocked and auth codes for the move were a snap with register.com – they were efficient, friendly, knowledgeable – and it took about five days. Covad was a nightmare. Total time – five weeks and multiple escalations. During that time Covad managed to completely screw up the zones too.

Step three is mostly complete too. Only one blog site to move – and the photos are uploading right now. This is really my frustration with GoDaddy. They have pretty good (i.e. I get what I pay for) hosting and infrastructure – but some of the grid hosting limitations and the associated responses from support are really frustrating.

The GoDaddy issue is that they either cycle the grid hosts (so an ssh/scp session is terminated) or they kill long running processes. With four photo blogs – some insane number of photos – total of some 80GB of data to move – I had to get creative.

Firstly copying the data via non-secure ftp wasn’t really my idea of fun. I started off with scp – but the remote host kept killing the connection. Next I tarred up the needed files – and the connection was killed. The final working solution to get the pictures up to GoDaddy was the convoluted tar – md5sum – split – scp – cat – md5sum  – untar. Moving 80GB in 200MB chunks with a retry script at my end was not fun.

The next issue was actually untarring these enormous tarballs. The first site unpacked just fine; the second kept being interrupted – i.e. tar was getting killed. There is no ‘nice’ on the server – so no way to fly below the radar. Turns out there is a process time limit of something like 180 seconds. This means that the practical limit to untar is about 13GB in size. My frustration with GoDaddy support was that they kept telling me to use ftp and that there was a limit of 100MB for tar. I spoke to GoDaddy support right at the start of this process and offered to PAY to ship a USB drive with the 80GB of tarballs to an admin to dump onto my space. I’d say there’s a value add here for GoDaddy.

Lessons learned:

Change control and planning are king. See previous posts. Nothing went wrong – but there were things that could have been smoother. What I guessed was a few weeks turned into a two month project.

Test with real-world datasets. Migrating a test blog with 200 photos isn’t a valid test.

First line support people often repost from the knowledgebase. A limit of 100MB for tar is unrealistic. Tell people it’s a time-related kill rather than a size issue. We can figure it out and workaround it.