Category: Life: Work and Techy

Sunday night/Monday morning: I didn’t sleep very well so I didn’t get much sleep. Therefore looking forward to Monday night. Unfortently, I’m “on call” until 11pm, but I should be able to stay awake until then.
Monday 10.50pm: We start receiving issues via our online Kayako helpdesk that some people are having database issues. As I don’t want alerts beeping me at night, I start investigating.
Monday 11.15pm: There’s a lot of Linux quota checking scripts running. Most odd – the maximum I normally see is 1 – and there’s pages full here. And I can’t “kill -9” (terminate) them faster enough. Only one thing for it – server reboot.
Monday 11.30pm: Server hasn’t come back up yet. Hmm, perhaps it just stalled on the reboot. Try again.
Monday 11.45pm: Not good, it still hasn’t come back up. I’ll have to alert the datacenter for investigation.
Tuesday 12am: Data center tries a reboot with a console attached
Tuesday 12.30am: Server is undergoing a file system check (fsck)
Tuesday 1.15am: Still undergoing the fsck. This normally takes around 20 minutes, but the datacenter is reporting lots of “inode faults” being reported on the /var/ parition on the server
Tuesday 2am: Still waiting for the check to finish
Tuesday 3am: Datacenter states it is unlikely to finish, but we agree to give it an extra hour
Tuesday 4am: I ask the datacenter to stick in a new hard drive and mount the old one as a slave.
Tuesday 8.30am: I’m already in the office having had less then 3.5 hours restless sleep. I start copying over the data from the old hard drive
Tuesday 5pm: Still copying, but only an hour to go
Tuesday 6pm: Most of the data is copied and no corruption
Tuesday 7pm: Nooooo! The entire datacenter goes down for nearly 45 minutes – when I had 6 SSH windows running to the server
Tuesday 8pm: Looks like everything is working, but let’s wait for customer responses.
Tuesday 9pm: Hmm, the email system doesn’t appear to be working correctly. It appears the the cPanel system can’t decide whether to use the “old mbox” format for emails or the new maildir format. The data from the failed hard drive expects to be read in the old format.
Tuesday 10pm: Email appears to be mainly working
Tuesday 10.30pm: Nope, still got minor problems with IMAP authentication
Tuesday 11pm: All is checking out and responding.
Tuesday 11.30pm: I head home
Wednesday Midnight: I’m home!
Wednesday: 0.15am: I expect to have dinner
Wednesday 1am: I expect to be able to fall asleep
Wednesday 8.30am: Gotta be in the office again.

I’m going to be a sleep-deprived zombie by the end of today (Wednesday!)

Yesterday, at the end of a 2 hour driving lesson, I think I left my mobile phone in my driving instructors car. Unfortently, it hasn’t yet “turned up” so I’ve had to report it lost to my mobile phone network – o2. And I was very impressed with their service:

• Only two menus deep with their automated system (press “0” for anything else or lost phone, press 1 to report a phone lost/stolen)
• UK call center with very fast answering
• Simple authentication steps (well, if I could remember when I last topped up my phone)

But the most impressive thing? They are willing to send out FREE of charge, and within 48 hours, a brand new SIM card with the same mobile number AND my credit transferred over. This is a on a “pay as you go” service.

This compares with around 4 menus deep, slow answering Orange (I think I was “on hold” for over 30 minutes) who I had a contracted phone with – plus they wanted to charge me £25 for a new Sim card that would take 14 days to be sent out.

So in summary:
O2 – I had no commitment to them. They were willing to send out quick and for free a new SIM card. My monthly spend with them is £15.
Orange – I had a rolling contract with them (for nearly 3 years). I had a monthly spend of around £25. They were going to charge me for a new SIM card.

So if you ever think of losing your phone (including the number of your driving instructor who /may/ actually have the phone!) – then go with o2!

Recently, we’ve made a few changes at work which, in theory, should lead to a less stressful job for myself. However, to allow these changes to get in place – I’m being even more stressed 🙁 I’ve having to keep an eye on 3 new recruits, along with doing 2 server transfers (one from a slow datacenter which has taken over 24 hours to copy over all the data – compressed!), setting up new servers, selling new servers, supporting an existing client base of around 20,000 customers, liasing with suppliers, writing a new billing system, writing a specialised ordering system, answering phone calls AND having to contend with the damn workmen outside the office window! Grr.

I hope it’s all going to be worth it!

Since I switched over from MovableType to WordPress, I installed the Search Meter plugin to find out what people were searching for – and the most popular search term is “boss” (with 10 searches so far).

Why, I’m not sure (I hope my real boss isn’t that paranoid that he’s searching for any references to himself here 🙂 ), but at least it gives me a chance to mention another site I’m involved in.

Behind The Front Line is was a user contributed site of “tales of woe from customer support personnel” and whilst I’m just hosting and running it for a friend, I’ve still made a number of contributions to it.

Previously, BTFL was running on Joomla/Mambo, but I decided on the server upgrade to switch it to a WordPress install as it feels more “suited” to WordPress. However, I had to manually copy of all the data from the Joomla database and then set up authors etc as WordPress hasn’t got a “Joomla import” option.

But feel free to have a look at Behind The Front Line and submit your own teeth-gnashing stories!

I’ve now got two spare hard drives “for sale” (for sale in quotes as I’ve got no idea what second hand hard drives are selling for) – perfect working order: I just had to replace the 120Gb and 40Gb drives with 2×20)Gb as I needed even more storage capacity. My main machine now has 3x200Gb HD’s in it (600Gb) and a 60Gb drive (that’s the machine’s original hard drive with Windows XP on it etc etc). Poor Zaphod is still powered down as it’s main boot hard drive (with RedHat Fedora OS on it) has failed and I’ve still got to hook a monitor to it to access it’s 400Gb RAID0 configuration and 200Gb additional drive.

Woh! That’s 1,200Gb (1.2 Tb) of “on demand live storage” I have. By June, I will definitely need to purchase some sort of backup solution – but has any body got any recommendations? I’m looking to have around 2Tb (2,000Gb) of on-demand storage by that time, so I’m looking for at least 3Tb backup space. DVDs and CDs just won’t cut it (given a DVD can store around 8Gb, I’ll need 375 DVDs and 5,000 CDs would just be silly).

So – anybody got any recommendations for a good 3Tb backup solution?