Network Status News & Information

This email is being sent to all clients on our UK based VPS servers about downtime due to occour on Saturday 2nd October 2010 between 00:00 and 08:00 GMT.

We expect our VPS servers to be powered down for between 4-8 hours whilst power equipment within the datacentre is upgraded.

Your VPS will be automatically shut down before the power outage and rebooted once power has been restored to the data centre.

If you have any issues accessing your VPS after this maintenance window, please contact us using the client area.

Below is a detailed explanation of the sequence of events from the point that the hard disk failed up until resolution.

At approximately 5am GMT, the server started reporting SMART disk errors, we were aware of the potential problem by 5.30am GMT and began to analyse the situation. At around 8.30am GMT, the hard disk failed completely preventing access to any administrative controls.

By 10.30am GMT, we had laid down a plan of action to follow in order to restore services as quickly as possible and had initiated contact with the data center. At this point we made the decision to upgrade the server to implement a RAID-1 disk array to provide a further layer of protection against a similar failure in the future.

The hardware upgrade had been completed by 1.00pm GMT, and we began to re-install the operating system and cPanel as quickly as possible. The cPanel installation completed at 3.30pm at which point we immediately started configuring it to be ready for us to start restoring user’s accounts. At approximately 4.00pm, we started to bring the backup files over from our backup server.

At 4.40pm, we started restoring the backups that we had brought over (with an ETA of 1-2 hours) and completed the restores of the majority of accounts by 6.10pm. From this point onward, we worked closely with users to rectify any issues that had occurred during the restoration process.

We apologise for any inconvenience caused by the outage as a result of the hard disk failure and are grateful for your support, understanding and patience throughout the restoration process. If you are still experiencing any problems with your website(s), we urge you to contact us so that we can resolve them as soon as possible.  

Dear Customer

First of all, we would like to apologize for this outage. In our few years, this is the first time that we have had a problem of this scale, and as such were not as prepared as we could have been. Rest assured that we have learnt our lesson from this episode, and have made plans for a new backup system to be put in place.

The reason for the outage was a zero day exploit against some server management software. More information on the attack can be read here http://www.theregister.co.uk/2009/06/08/webhost_attack/.

Over the past two days we have been in contact with our datacentre trying to resolve the issue. This morning, it was announced that the data on all three of our servers ( Two in the UK and one in the US ) had been lost. Following this news, this morning we purchased a new dedicated server at the BlueSquare Datacentre ( Maidenhead, UK ) and restored all of the backups that we had. Please note that our terms of service state that "Customer is responsibile for maintaining their own back-ups of their data in case of data loss from PrimeXeon's servers.". 

We have worked solidly since the incident trying to restore services as quickly as possible and we now have most of the backed up accounts restored. If you cannot access your account/domain(s), please open a support ticket with us by logging in at http://www.primexeon.com/clients/ and going to "Submit Ticket".

With regards to backups, our new system is going to be based on three points of defence. All of the data on our servers will be backed up into two separate locations (New York and Cambridge) via a scheduled backup on a daily basis. On-top of this, we will also be making complete account backups to one of the two remote servers on a weekly basis and for all accounts.

We are looking into a number of other systems and have intent to setup a status site which will be hosted in a datacentre separate from any of our servers. In addition to this, please read what we will be doing for you, our loyal customers, below:

 - We will be applying 1 month worth of credits to every customer that has experienced downtime

 - We will be applying 2 months worth of credits in case you have suffered loss of your data

Please open a ticket with us in either case so that we can add the credits to your account.

We apologise for the inconvenience caused and thank you for your patience 

Kindest Regards,

PrimeXeon

We will be updating MySQL on a number of our servers.