Outage 10/09/2015

[#1]

It's "fixed" they say
That would be news to me, my connection seems to just be chugging along, almost like someone is unplugging and then plugging back in an ethernet cable every 5-10 minutes....
Browsing the web is so stressful too, page's load one minute then take up to 3 minutes (I've timed so far to load), I've had a lot of downloading to do from Apple's CDN the last couple of nights... my poor iPhone didn't know what the hell was going on last night when it was trying to get the iOS 9.1 Beta... I ended up switching off the Strong 5Ghz WiFi and using a 2 bar 4G signal from EE that got the download from failing the download to zipping along and 1.3GB was downloaded very shortly after this move.
I had to hop gateway to get Now TV to buffer a live stream even though when I did tests on the connection I was still getting full speed on both single and multithread downloads...
I would normally be listening to the Radio on my Sonos system now, but after it also struggling "a lot more than usual" to get connections over the last week and being very choppy and dropping I've given up and opted for whatever's on the NAS.
I feel that Plusnet, Have truly made the last week a royal pain in the arse for this household, and with one week left until migration I can't see it getting any better.... roll on peak Sunday I suppose

[#2]

Quote from: ricke17

Touch wood, I was not affected.  This is interesting though: http://www.thinkbroadband.com/news/7156-problems-over-at-plusnet.html and here: http://www.theregister.co.uk/2015/09/10/major_plusnet_outage_dns_goes_titsup/ I don't use pn DNS so I don't know if this is why I was not affected. Rick.

I do use Plusnet DNS (at least that's what my router is currently set to and all my devices use that for DNS), I was unaware of the issue until now and can't believe that nobody in the house was using the internet during the 'outage' period (they'd have definitely said something if they had a problem).  It appears that I missed the outage, anyone else?

[#3]

I wasn't affected and my routers set to use Google DNS from the WAN.

[#4]

Quote from: Trying

@ lorisarvendu Whilst I accept everything you said about the complexities of running any of todays essential services ie electricity, internet. etc., I think what has caused the most frustration within us Plusnet users, was the total lack of information and feedback, to what can only be described as a major outage. We do not live in a 9-5 society anymore, but a 24/7 one. If people were in fact working on the issue to restore services from the outset of being aware of the problem and its significance, why couldn't someone at Plusnet have at least acknowledged the problem and advised as such. Why were the first words from Plusnet not uttered until approx 07-50 am this morn? Hopefully the support staff will give a more detailed explanation in the very near future, such that those with more than just consumer interest can understand what actually happened.

I totally agree that PN's comms could have been better, but I think the overall reaction to this (everywhere, not this board) has been excessive.  Forums everywhere seem to be full of posters spewing their hate for Plusnet, all because they didn't let us know in time what was going on.  We don't know what caused this.  It could have been component failure, or an outside attack, or somebody at PN making a stupid mistake, but whatever the reason was, PN may well have fixed it in the quickest time they could.  I don't believe the whole of Plusnet just sit on their backsides raking in the cash and laughing up their sleeves at how they're screwing over the punters, yet you'd think that was the case when you read some comment on the Daily Mail about how "my internet's been done all morning, Plusnet are $&~£%!"  I bet there were people working 20-hour days to get this sorted, but nobody thinks to thank them. 
I guess if that attitude bugs me, I shouldn't really be reading an internet forum, should I?

[#5]

Quote from: tijara33

Speak for yourself Andy, not for the rest of us who would love to know the reasons for yet another PN cock up.

If you don't know the reasons, how can you be so certain it was a PN cock up, tijara33?  If your car breaks down, do people refer to it as your cock up?  Or do they consider that it could just have been a component failure?  If it was a router or switch going pop, then it might be more correct to call it a Cisco cock up.

[#6]

I'm wondering if everyone affected was on a pcl gateway and those unaffected were on ptw or ptn gateways?

[#7]

Quote from: lorisarvendu

If you don't know the reasons, how can you be so certain it was a PN cock up, tijara33?  If your car breaks down, do people refer to it as your cock up?  Or do they consider that it could just have been a component failure?  If it was a router or switch going pop, then it might be more correct to call it a Cisco cock up.

An equipment cock up can be looked at in many different ways too, if, for example, it was a 20-year-old sellotaped piece of gear that was used due to lack of investment. So whose cock up this would be?
Since Plusnet are not providing any explanation and unlikely to so, any speculation can potentially be true.
I appreciate you defend honest and hard-working Plusnet staff and engineers (who doesn't?) but the general feeling of the things happening at Plusnet right now is that of achieving sales at all costs and that is having an impact right now on the customer service, and the service in general (like peak-time slowdowns/capacity discussed on these forums in great detail, and, lately, a couple of mysterious outages in a very short period of time).

[#8]

Quote from: spoon

I'm wondering if everyone affected was on a pcl gateway and those unaffected were on ptw or ptn gateways?

Pretty sure I'm on pcl-ag02, router reports gateway as 195.166.128.183 and not affected as far as I can tell.

[#9]

Quote from: artmo

I wonder if we will get an apology from Andy Baker, CEO of Plusnet. Like all CEOs he was asked to introduce himself when he joined PN but I can't remember him doing so. He has certainly not involved himself in the forums.

You want the CEO of a major UK ISP to come on an internet forum and apologise for the fact that his technical team diagnosed and successfully dealt with a major outage (that you don't know the cause of)?  Or the fact that his company didn't update the Service Status Page?

[#10]

Quote from: alext05

An equipment cock up can be looked at in many different ways too, if, for example, it was a 20-year-old sellotaped piece of gear that was used due to lack of investment. So whose cock up this would be? Since Plusnet are not providing any explanation and unlikely to so, any speculation can potentially be true. I appreciate you defend honest and hard-working Plusnet staff and engineers (who doesn't?) but the general feeling of the things happening at Plusnet right now is that of achieving sales at all costs and that is having an impact right now on the customer service, and the service in general (like peak-time slowdowns/capacity discussed on these forums in great detail, and, lately, a couple of mysterious outages in a very short period of time).

Speculation is fine, when you don't know the cause. But speculation on this board frequently goes into the territory of attributing ulterior motives to the Plusnet team with no evidence whatsoever.  You've just done it yourself above.  "Mysterious outages"?  Are you seriously implying that Plusnet engineer these outages to help their unproven goal of achieving sales at all costs?  On what planet does showing that you are providing an unreliable service generate sales?
Peak-time slowdowns and capacity are affecting every UK ISP. You only have to look on other forums.  BT customers are complaining.  Virgin customers are complaining.  Sky customers are complaining. This isn't to do with PN reducing capacity and customer support, it's to do with the increasing customer take-up of broadband in this country and the creaking BT infrastructure its expected to run on.
I dread to think what a Plusnet engineer who has just worked his nuts off to get services up and running would think if he looked on here and read some of the vitriolic reactions to his work.  I bet that if PN found out that the cause was completely out of their control and could be neither anticipated nor prevented, nobody on this board would thank them or apologise for earlier comments.  It would be more of the usual "Yeah, well they fixed it this time, but they're still rubbish!"

[#11]

Quote from: tijara33

Quote If the internet is so critical to your income you would have failed over to your backup provider no? Oh, so it's my fault that PN do not monitor their systems overnight, do not have any engineering support & rely on their customers to find faults in their system? So to cover their failings I'm supposed to shell out even more of my hard earned cash? No, like many others, once my contract is over I'll be leaving. Maybe this is what PN desires: reduce their customer base to allow their totally inadequate systems to cope.

No it's not your fault Plusnet had an outage, that's largely outside your control.
But what is your fault is you have a consumer/basic business service with very little in the way of SLA (if anything), therefore if it's critical to your business either pay through the nose for a service with a guaranteed SLA (at which point you'll at least get compensated for the outage even if it ends up being pittance) or have a backup of some sort in place.
DSL lines are cheap so having a 2nd one with another ISP is a good step to at least get some resilience, it still won't help if the local exchange goes up in smoke or someone shoves a digger through the duct but you are at least protected from a single line fault (Which btw BT can take days to fix) or one of the ISP's going TitsUp.  If people really are as they claim losing £100's due to an outage then paying less than £500/yr for a backup cct is a no brainer.

[#12]

Quote from: lorisarvendu

Speculation is fine, when you don't know the cause. But speculation on this board frequently goes into the territory of attributing ulterior motives to the Plusnet team with no evidence whatsoever.  You've just done it yourself above.  "Mysterious outages"?  Are you seriously implying that Plusnet engineer these outages to help their unproven goal of achieving sales at all costs?  On what planet does showing that you are providing an unreliable service generate sales?

I think you might need to re-read what I've written, preferably after a brief period of meditation. Nobody is implying that they are engineering them. Heaven forbid! Would "unspecified outages" term be better for you? You know, the ones that no one knows the reason for why it happened?
And to make it clearer as to what "sales at all costs" means, I would like to suggest that it primarily means lack of investment in areas such as: a) capacity; b) customer support; c) hardware; d) training; e) recruitment, ect, ect. It also usually results in: 1) increased expenditure in advertising; 2) free broadband deals left right and occasionally centre; 3) extremely generous retention deals.
And please stop going on about hard-working Plusnet engineers working their nuts off. No one blames them, for goodness sake! When people blame Plusnet they either blame the folks at the top making those hard-reaching decisions, like reducing customer support hours or preventing investment in needed areas, or, occasionally, customer support for messing things up (either human error or system error - Plusnet provisioning is known to be iffy).
The main complaint yesterday, by the way, was about lack of communication. And you don't have to be a network engineer to understand that.

[#13]

Quote from: lucerne

The moans about lack of communication remind me of a saying that went something like, "when you find yourself swimming with sharks, the last thing in your mind is draining the ocean". Let the guys get on with it. I may be very well in a minority here, but have my doubts how many PN punters would gain very much by knowing what the cause of the malfunction was.

You have managed to completely miss the point. Posting that there is an issue somewhere where their customers will look, will immedately tell said customers who look that the problems they are having are NOTHING to do with their particular connection, and THEN they can let the guys get on with it, not start messing around with their own perfectly functional setup, or attempt to log thousands of tickets or phone calls.
It is entirely beneficial to Plusnet's interest to tell people the fault is a general one, and not a connection specific one. It is only co-incidental that it is also beneficial to their customer's interests too.

[#14]

Just to conclude my contribution, my 'uncontactable website' issue was resolved at 13.40 today - after about 36 hours or thereabouts.  
This was following a lengthy text discussion with its tech dept. and an analysis of my static-IP was scheduled for midnight tonight, through to 7am, tomorrow.  The website became contactable, a few hours after the discussion, so I have given PN the option to cancel the overnight tests, unless they feel they are still useful.  I am yet to learn (if I ever do) what they may or may not have done to correct it, between the discussion and now.  I suspect it was just the DNS servers catching up and synching, which resolved it, but that is speculation.
I don't care much for apologies - I consider them pointless - and would rather they just do better next time.  I would have left them to it, and PN would have saved me hours or exploratory work, if they had just said, "We have a problem that might be causing some of you not to be able to access some websites.  We are working to fix it."  I would have been satisfied to get on with other things, if I have heard/read such a statement.  I would only have become involved, once the problem was claimed to have been solved, and I still had problems.  However, even then, I would have known where to go to seek a resolution, rather than making hours of superfluous network, browser and router changes, bottleneck/common-factor elimination checks, together with all the associated web searches.
I consider that PlusNet provides an excellent service, with (in my experience) excellent, responsive support staff.  However, this reluctance to immediately declare issues (even if they are unknown) causes vast amounts of unnecessary work and hassle for those of us who do not just sit back and wait, hoping some invisible friend will solve the issues.  This does not protect the PN reputation, it damages it, as this lengthy thread - considering it is little over a day old - clearly demonstrates.

[#15]

Understandably there has been much said about the general lack of communication about this failure.  A number of suggestions have been made, a number have merit, but (I suggest) in a the case of a major outage have limited benefit.
The first line of enquiry about an issue is the call centre: for residential users this now closes at 10pm.  On discovering the issue around 01:45 so as to ensure that it was "known about" I just phoned business support and told them that there was a network problem.  They claimed no knowledge of one and assured me if there was one "the networking boys upstairs would be monitoring it".  They did say though that they would pass a message on to them.  One hopes that at that point someone looked, found a problem assessed it and initiated a serious incident alert, the actions from which ought to have been.
1. Recorded message on the support numbers advising of a major incident under investigation
2. Recorded message placed on the status phone line
These measures are useful and effective assuming that calls route to the phone management system.  I understand that the incident also brought down the phone system so even if that had been done, that line of communication would have been rendered useless
3. A service status update is posted on the user portal - its a good place to advertise matters if it is accessible.  I understand that too became inaccessible both for users to read and for staff to post status information, rendering that useless too.
4. A service status be posted on these forums.  Strangely enough this non-essential business platform is the one part of the PlusNet offerings which has stayed on air throughout most of the major incidents in the last 2 years.  It is however used by only a small proportion of PlusNet's user base, so cannot be considered to be a primary communication channel, albeit one having a certain following and high availability.

It has been suggested that the status be posted on a third party site - setting aside the "Is PlusNet really going to publish a major systems failure notice on a third party site?" question - if users cannot access the internet from their PlusNet connection how are they going to find / read that message?  Not everyone has an alternative means of connecting to the internet.  Remember for this incident switching to other DNS servers did not fix this issue for everyone.  Fixing DNS resolution issues does not address a routing failure inhibiting access to a target service.  Depending on the nature of the issue (even with an alternative DNS) users might not be able to route the such a third party site.
Some communication is better than no communication.  I would have hoped that from all of the BNG failures about a year ago, PlusNet would have understood the necessity of clear communication around major incidents.  It needs to be prompt and not dependent on key staff (only CRT staff) who are not available 24x7.
Notices ought to be placed on all of the recognised points of contact - main phone lines, status line, user portal, this forum and then if thought appropriate third party communication channels - assuming of course that they are accessible.  If these services are not accessible over PlusNet's internal networks, then consideration needs to e given to going into the machine room and configuring status reports via the server's platform console!
After what might reasonably be done has been tried, there after we'll have to accept that sometimes a failure is so pervasive that the systems which administer all of these communications channels are simply not accessible to facilitate communication.
As for fixing the problem, there is nothing worse than a partial failure to try to understand what is actually broken.  The experience of most - some sties accessible and other not does rather take some working out - "Why some and not others" - there has been lots of speculation but sadly no facts.