Outage on 12 May - lessons to be learnt

[#1]

When I first tried to get on on 12/5 I assumed it was a local problem but soon discovered it wasn't. I rang Plusnet and (after having to enter my phone number twice) I learnt about the problem - next update 11 o'clock. Rang again about 11:45 - same issue with telephone number - update "later this afternoon". Rang about 17:30 - telephone number twice again - problem is cured but you might have to reset your router - which I did have to.
According to BT fault 15904, service was fully restored at 10:45, so what happened?

[#2]

Thanks for the feedback. There are lessons to be learned here. Mainly about how these issues are communicated internally and how visible they are. From what I can see here the problem was the agent at 11:45 may not have been aware of the most appropriate information to update you with.
Adam

[#3]

I did not actually speak to a person. These were recorded messages! If I remember correctly the message I picked up at 11:45 stated that it was updated at 11:30!
Also, having spent much of my career working with customer facing systems (ATM & credit card authorisation) I find it unbelievable that there is a single point of failure like this. Was it that there was no way of routing round the failing component or was it that no one knew how to do it?

[#4]

We're not fully sure what actually happened, only that we (and of course our customers) were affected by three separate supplier outages that day.
I'm not sure which bit went that caused your issues in particular and am sorry that we can't give you any more information, however as Adam says we've fed back on the comms that went out and will be ensuring they're improved should we end up in a similar situation in future.

[#5]

Frightening! That may be good honest Yorkshire, but it doesn't fill me with confidence for the future.

[#6]

Quote from: ponyman1

it doesn't fill me with confidence for the future.

Every ISP is in the same situation!
There are going to be outages that will affect all ISP's including LLU, That is the nature of the internet!

[#7]

I agree with ponyman1. Whilst there will be single point failures at a local level because it's far too costly to provide appropriate redundancy, that should not be the case where major parts of BT's national network is concerned.
A single point failure that puts 100's of thousands of users off-line for hours on end is totally unacceptable. And this is not the first instance of such failure either - at least 3 to my knowledge in recent times IIRC - I'm not going to waste time searching them all out.
OK, perfect and automatic redundancy may be tricky, but it should be possible to deal with a number of situations within minutes by simple software manipulation of routing (not the bodged job that took them over an hour on the 19th April) - this may lead to some congestion for a while until the outage is fixed, but at least users would be on-line.
BT clearly need to go away and learn some basic Reliability Theory and then put it into practice.

[#8]

I agree with what you say Anotherone!
However, I was trying to point out that while BT controls so much of the UK network any outages on the BT network are going to have an impact on all ISP's even those who are LLU! 

[#9]

The problem is that, in spite of OFCOM et al, BTW is still a monopoly supplier and will do what they feel like with no comeback for anyone. I well remember a situation in the 1970s when my employer who had two leased lines to the Channel Isles and paid a premium for diverse routing, suddenly lost both of them. BT (or was it the GPO in those days?) apologised that a dredger had caught a cable in the Solent. What about the circuit from Portland we asked. Oh, that was moved on to the Southampton cable a long time ago. With any other commodity you would move to another supplier but with them you couldn't and still can't.