The Useful Arts Net

Over the coming months I'll be putting to bed and transferring it to, bringing all services under the roof of The Useful Arts Organisation

Web Server Failure

On Thursday the 25th at approximately 20:42 a power failure with another server in the rack cause a breaker to trip, powering down my server. Initially the server seemed to come back fine, but errors were later spotted. These were due to a physical error on one of the hard drives where all of your websites are hosted, causing the disk array it is on to fail. At approximately 23:48pm the server was rebooted in an attempt to get the drives back on line, and again failed to recover fully.

My ISP were informed of the issued and worked through the night, but were unable to recover the hard drive arrangement. I started a recovery of the data from backup to a second disk array within the system with the aim of bringing up the core of the web sites in as short a time as possible - though the 146GB of data would have taken a long time to copy up!

In the meantime I also tried to recover data on the second local drive - the drive which had failed was one of a pair storing all website data in parallel, and the secondary drive was also failing to recover for unknown reasons.

At around 17:30 yesterday I managed to recover and get access tot the data on the spare hard drive which had failed, and proceeded to copy the data from that to a new pair of master disks. This was completed at 21:45 yesterday evening, and full service was restored. As far as I am aware no data was lost.

The system which went down was purely used for website hosting, and so none of your email was affected during the outage. Mail Problems

You may have noticed some intermittent email access for the last few hours. This is due to a mail server at Demon Internet which is attempting to deliver the same message over and over again, tying up all of the server resources.

I have contacted both Demon and the originator of the email message (which is not spam) and neither of them have been helpful in resolving the issue as yet, Demon saying it's the customers problem - and the customer must log a problem with Demon - and their customer has not yet responded to my email, and I've been unable to track them down on the phone.

So, in the meantime, I've had to block all mail from Demon Internet, until such time as I can be sure the problem is cleared. This may be for up to 48 hours. Allowing all this mail from Demon will mean the server is likely to fall over, or other mail will failed to be delivered, and so this is the best course of action.

I apologise for any inconvenience this may cause, but unfortunately it's outside of my control. I will continue to try and resolve the issue with the other parties.

... and all that jazz

A warm welcome to Simon Rose who's come to live on the HostEverything servers.

Further Outages - This time a network issue

Availability of the server was a bit poor yesterday due to a network failure at our ISP's datacenter, BlueSquare... details from my ISP below:

Sorry for any inconvenience this caused. I'm speaking to my ISP how they can improve their redundancy.

The connectivity to this server was only lost to the UK for the majority of the time, with network peering from the US (and thus Asia and Australia) and Europe remaining OK.


Dear customer,

Today we experienced an issue with BlueSquare's primary fibre link - a
break in the fibre was detected at the Telehouse (East) datacentre in

Traffic is currently being routed through BlueSquare's secondary fibre
link, whilst engineers work on repairing the primary fibre.

During this time, however, there was significant packet-loss due to the
third fibre connection between Telehouse (East) and Telecity (HEX). This
is what caused the majority of the outage that customers, PoundHost, and
other providers in BlueSquare experienced this afternoon.

At this time, traffic is still being routed through the secondary
connection, although extensive work is being spent to bring the primary
connection back in service.

Let me reiterate that this issue is not directly related to PoundHost -
we are one of a handful of providers at BlueSquare affected by this outage.

We will send out a more detailed announcement once we are confident that
service will no longer be affected.

Many thanks for your support and patience during this time.

Update: 03/02/2007 10:51 and this just in... a further explanation:

Further to the two e-mails sent yesterday, I can now confirm further details of yesterdays issues.

The main cause of the outage for PoundHost, and the outage suffered by other ISPs located in BlueSquare, was due to their fibre provider, TeliaSonera, *disconnecting* the wrong fibre when testing another customer's link, due to them mis-labeling the fibre in their London datacentre suite (Telehouse East).

As per yesterdays email, our second diverse fibre, which was working fine, was down for general maintenance due to router and switch software upgrades which we were carrying out. When the primary fibre went down, we bought our second fibre up as quickly as possible, however due to the maintenance we were doing the connection caused packetloss and routing from some ISP's but not others.
We have now fully completed our maintenance on this fibre overnight, so in the event of any further issues on the primary fibre we can easily route over the second diverse fibre without the issues experienced yesterday afternoon.

Some customers have mentioned that contacting us via phone was difficult, this is due to our phone system being VoIP (voice over IP) based, so the bad connection meant we had troubles hearing customers, and in some cases no phones at all. I have this morning ordered 4 BT lines to replace the VoIP system to be installed within the next 14 days direct into our support office using the existing numbers, to avoid this happening in the future.

Following customer feedback, we have now setup a new NOC (Network Operations Centre) status page, which we will be finishing and launching on Tuesday to keep customers updated should there ever be any network issues in the future. This will allow you to get a quick glance status, and updates such as these on the website. We will email you on Tuesday once this site is completed with more details.

Again, we apologise for any inconvenience caused yesterday. It's an unfortunate chain of events that we already have systems and links in place to deal with (this kind of fibre break), but as with all things in life when you need it, it's the one time it isn't available, in our case the second fibre being down for our own internal maintenance. For reference, our two diverse fibres are called 'THE' and 'TCX' which you will see in any 'trace routes' you complete, depending on the routing destination.

Mail Outage : Update

After a night of hacking around and some hair-raising and hair-pulling moments I've done a full upgrade of the operating system on the server, to try and resolved some dependency issues in some of the Libraries. This has left a few other things out of shape, which I'm in the process of trying to sort out. Hopefully this will all be back up and running in a few hours (and then I can get some sleep!)

Sorry for the inconvenience caused and thank you for your patience.

Mail Outage

We're having some problems with mail running through at the moment due to a flakey SSL library causing problems with POP3 mail checking. I'm doing some heavy emergency upgrades to try and (hopefuly) get everything back into sync asap.

Server Outage 12/11/06 11:30-16:10

Earlier today there was an outage when one of the disks in developed a fault, and had to be rebuilt. No data was lost in the process, but the server had to remain off-line during the rebuild (which takes around 3.5 hours to run).

The server is now back up. All undelivered mail has been queued on our backup mail servers and will start to find it's way into your account in the next hour.

Sorry for any inconveience caused in this outage.

Scheduled Downtime: 17 Aug 2006 20:50 GMT for 3 hours

Just a quick note to say that this Thursday, 17th August 2006, the primary server (suith) that your email and website is hosted on is going to be moved to a new hosting facility. The server needs to be physically moved from one side of London to the other in the process so your email and website won't be available for a period of approximately 3 hours, from 20:50 GMT onwards. I know this is a bit of a pain but the result will be faster, more reliable connectivity. And I won't have to pay an 80% increase in hosting fees.

Any email that is attempted to be delivered during the down-time should be delivered once the service is restored, so the effects should be negligible to those on UK time. Sorry if it's a bit more inconvenient to those in the USA and elsewhere.

Network Downtime, Friday 28th July 7.30am to 7.50am

this might affect your access to services:

Dear Customer,

We need to perform emergency maintenance on our core routers in our Maidenhead Data centre at 7.30am tomorrow morning. [28/07/2006 - BST]
This will unfortunately cause approx 20 minutes of network downtime, which we hope can be reduced if all goes well to just a few minutes downtime, however we are allowing a 20 minute window for worst case.

No action is needed on your part and this will only affect Maidenhead hosted customers, London is un-affected.

We apologise for the short notice on this maintenance, however it is critical to continued smooth running of the network.



More Outages

Apparently there is a problem with the Fibre Optic connection where my server is hosted. hmmmmmm.

Outages: Network Failures

In the last few days there have been a couple of outages which have affected the main web and mail server. The first was on Friday the 31st of March and lasted for about an hour between 17:30 and 18:30, the second on Monday the 3rd of April between 13:30 and 16:30 . These outages were due to network problems at our UK ISP, PoundHost and they are working to make sure it doesn't happen again.

Sorry for any inconvenience caused.

East Side vs West Side...

A belated welcome to Bethnal Green art gallery Hotel and Fulham gallery West London Projects

New Family

welcome to Stylofoam, BCN Sound and Urban Logo

Service Resumed

Just to let you know that the maintenance scheduled for today has been completed. has a new primary disk drive and the live backup is fully operational. All services have been restored and recovery in the event of power failure has been tested.

Unfortunately during the swap some old email was restored due to the backup that I built the new drive from. Instead of risking loosing any new messages I've left this as is, but you may find you get some duplicates of messages received earlier in the week. Sorry for the inconvenience.

Schedule Downtime 17/12/2005

Further to my message earlier this week I'm going to be taking the hosteverything server down to fit replacement hard drives on Saturday 17/12/2005. I'm aiming to start work at 9am, and hope to be finished by 1pm. It might be considerably quicker than that, all being well.

Site Launch

Given the failure in the last days I've set up this page which will be used for status updates and information for all hosted sites. What's that you say, if the servers down you won't be able to see this page? Yes, indeed, that's true. But I'll get to that later. I have a plan.

Unexpected Server Outage 10/11 December

I'm sorry to report that after a smooth switch of's server to a new ISP and server with more space and bandwidth, back in August, that went offline on Saturday the 10th at around 16:00.

Initially I thought it was a network outage - there had been a brief one earlier in the week - and a report was filed with my ISP. The network was reported up, and then the server responded to being up but no longer accepting connections from other systems. The server was rebooted but failed to become available - it was on-line but not accepting connections. Upon visiting the hosting facility I discovered that the server had come back up, but was coming on-line very slowly due to an email problem - the server was being flooded by incoming email. This was resolved and the server left running. At approximately 1:30 on Sunday the 11th the server went off-line for a second time.

A second visit, the following morning, isolated the problem was with a faulty network cable that had been further strained by the previous visit. This was replaced and the server became available at 10:30 on the 11th. I decided to check the server was starting up properly, and upon reboot a faulty hard disk was discovered, which was also slowing the system from starting up. The drive was removed for testing & replacement.

No data was lost during the outages, though a few pieces of mail - mostly spam - may fail to be delivered most will have come through. An additional offsite backup of the live system was taken at 11:30am on the 11th.

A replacement hard drive for the faulty unit - a 'live backup' of the main hard drive in the server - was purchased. This was installed in the server at around 15:00 on the 11th, taking the server offline intermittently within the following 2 hours. During this process I discovered the new drive was 41Mb smaller (grrrr) than the required size and could not be used as a mirror of the live volume. Instead a snapshot backup of the server was taken onto the new drive and it removed.

The server has been on-line since 16:10 on the 11th and is running to pull performance. Incremental offsite snapshot backups continue to be taken nightly at 01:00.

The new hard drive is currently being set up and tested on a second system and I will hopefully install it within the next week. I'll send out a notice to let you know when to expect downtime, but I'm afraid it's looking likely to be around 4-5 hours of downtime whilst all data is backed up, the new drive is switched in (as live drive) and the old live drive is moved to live backup, all tested and the system brought on-line. I'm sorry for any inconvenience this may cause.

Let me know if there is a particularly bad time for the server to be taken down and I'll schedule to avoid that. It's already scheduled for outside of office hours.