Outage 10/09/2015

[#1]

Would it not make more sense to move it to another physical location, may be on BT's platform, so it can at least be accessed in the event of downtime. If the outage is that big that BT are offline then it would affect more than Plusnet
Corrected typos.

[#2]

Quote from: KevinAndrews

Loving the quote of ISPreview.. At present it’s somewhat less “broadband that loves you back” and more “broadband that doesn’t work properly“, lol http://www.ispreview.co.uk/index.php/2015/09/website-and-connectivity-woes-strike-plusnet-uk-custome...

The outage has now also made it to The Register: http://forums.theregister.co.uk/forum/1/2015/09/10/major_plusnet_outage_dns_goes_titsup/

[#3]

Hi guys thanks for your patience.
Since early this morning we’ve experienced an outage which affected our phone and broadband service as well as our website. We’ve worked throughout the night to resolve this and have made a lot of progress with most of our customers now up and running.
We still have a few issues which we’re working to solve as soon as possible. We would like to sincerely apologise to our customers for the inconvenience caused.

[#4]

Quote from: goldenfibre

I recommend someone to join Plusnet and it went activated two days ago and now they BLAMED me! Thanks you very much Plusnet. U have cost me a friend loss!

But it's ok now isn't it.... according to what you have said over at ThinkBroadband you have lied to them to keep them sweet, and also referred more friends to PLusnet so that you will soon be paying nothing for your Fibre Extra service....
So I think its fair to say, don't complain about something being provided to you at the costs of other customers!

[#5]

Quote from: dragon2611

Quote from: tijara33 I'm back now but not being able to access the internet since 06.00 hours has cost me a small fortune. Well done PN, thanks for nothing! If the internet is so critical to your income you would have failed over to your backup provider no?  I'm afraid I'll never understand why if it's that important people don't have 2 connections with different providers as a minimum level of redudency

I mean, surely for £15 a month we ought to be supplied with 24/7/365 Zero-downtime systems with seamless failover for our mission-critical viewing of cat videos on YouTube?
I mean, COME ON PN!
[sigh]

[#6]

Certainly not, but an update to the Service Status or Twitter or the Forum wouldn't have gone a miss.
dick:quote

[#7]

This^^  A little information goes a long way. Very poor public relations, worthy of BT... oh, wait...
dick:quote

[#8]

All our services are now fully restored, including the Plusnet website. Thank you for your patience, we’re really sorry this has happened and would like to apologise for the disruption you’ve experienced.

[#9]

Are you going to tell us what the issue(s) were, or do we just start the rumour mill and leave it to run?

[#10]

It's now hit the national press
http://www.telegraph.co.uk/technology/broadband/11856185/Plusnet-outage-leaves-thousands-without-pho...
http://www.theguardian.com/technology/2015/sep/10/plusnet-users-suffer-outages-dns-problems
http://www.theregister.co.uk/2015/09/10/major_plusnet_outage_dns_goes_titsup/
http://www.theinquirer.net/inquirer/news/2425351/plusnet-broadband-outage-means-it-cant-do-you-proud

[#11]

As someone who uses Plusnet DNS servers and hasn't had a problem is there an explanation as to why some are affected and some aren't

[#12]

Never let the facts get in the way of a good story...  The Telegraph article headline states that phones were out as well!

[#13]

As someone who has worked in a university IT dept for 18 years, I understand perfectly well why it has taken Plusnet so long to fix this.
When parts of a computer network become inaccessible it is very difficult and time-consuming to find out exactly where the fault has occurred, because your primary diagnostic tool is the network itself.  Imagine a blackout suddenly affecting a large part of a city. You know (or you guess) that somewhere in the middle of that mass of darkness is a blown component that has caused all this.  However you can't use any electrical tracing equipment to track down where the error has occurred...because there's no electricity.
It's the same with a routing problem.  You've got a whole area of network where none of your switches or routers are responding.  One of those switches or routers is the culprit, but there's no way of knowing which one because none of them are accessible.  You can't get any diagnostic traffic in there, because there's no way of routing it!  You are reduced to working your way methodically through the network, testing each component step by step until you find the one that's broken.  You then probably have to reboot all the other switches just to be on the safe side.  
At the Uni we've had to do this a few times in the past, and each time it took at least a day. We're one organisation.  Multiply the size of our network by a thousand and you still wouldn't approach the complexity of Plusnet, and what their engineers had to wade through (everyone was shouting DNS at the beginning.  Ironically if it had been DNS it would have been resolved far quicker - replace/restore the DNS servers.  Sorted.)
It doesn't matter how big Plusnet is, or how many customers they have.  There's no way they can guard against a major failure like this, short of having a complete duplicate of their entire network infrastructure as a backup, and that would be insane.  In fact the larger the network is (i.e. the larger the blackout in our hypothetical city) the longer it could take to fix it.
Well done Plusnet for doing your job to the best of your ability, amidst a certain amount of criticism from customers who know nothing about how IP networks and the Internet actually work.

[#14]

Well said @lorisarvendu 

[#15]

Quote from: lorisarvendu

Well done Plusnet for doing your job to the best of your ability, amidst a certain amount of criticism from customers who know nothing about how IP networks and the Internet actually work.

How patronising.
In fact, this thread is full of people complaining that PlusNet have not handled the communication around the outage correctly.  Which is totally fair, because it makes them look like complete amateurs, regardless of how professionally (or otherwise - how would we know when they tell us nothing) their engineers have behaved.
Frankly, as a PNer for 15 years the removal of 24hr support and lack of communication is a worrying sign.  With a home move coming up, I'm looking around at alternative providers rather than hoping PN don't mess up the changeover.