Packet loss on Unlimited Fibre Extra

[#1]

Hi,

I am currently experiencing high packet loss on my Plusnet Unlimited Fibre Extra connection. I have various network monitoring tools running continuously so I have a record and history of the network health. The problem started at 13:20 on the 27th March and has not shown any signs of improvement since.

My router (a DrayTek Vigor 2762) forwards all logs including the PPPoE and ADSL connection stats to a server I run on my LAN. I can see that there was no connection drop or reduction in line quality around when the problem started. In addition, DLM has not made any adjustments to the line indicating that the copper phone line is still good.

As a test on the 31st March, I installed the original Plusnet router again in place of the DrayTek and still observed a high amount of packet loss. I have run ping tests using both routers with all LAN devices disconnected and WiFi turned off. On the DrayTek, I performed the ping tests on the router itself to bypass any issues with the LAN or my devices but the packet loss was still observed.

Here is a result of MTR to Cloudflare DNS (1.1.1.1) performed over wired ethernet an hour ago:

Hop	Hostname	Loss %	Sent	Last (ms)	Avg (ms)	Best (ms)	Worst (ms)	StDev (ms)
1	VIGOR.router	0.0%	31	0.2	0.2	0.2	0.6	0.0
2	172.16.18.15	16.1%	31	3.9	4.9	3.7	13.8	2.0
3
4	128.hiper04.sheff.dial.plus.net.uk	10.0%	30	13.0	13.1	12.4	15.6	0.6
5	peer7-et-7-0-1.telehouse.ukcore.bt.net	23.3%	30	12.4	27.7	10.2	67.6	15.6
6	109.159.253.95	20.0%	30	15.2	16.4	13.2	26.8	4.0
7	172.70.94.4	24.1%	30	14.4	16.0	13.9	25.0	2.9
8	one.one.one.one	13.3%	30	12.0	12.9	11.2	22.1	2.5

As part of my network monitoring, my server runs MTR every 30 minutes and saves the results into a database. Before the problem started, both the 2nd and 4th host would almost always respond but now the packet loss is ranging between 10-20%. As shown in the table, I am also seeing packet loss to Cloudflare DNS. I am seeing the packet loss to every host which I test against on all protocols, not just Cloudflare/ICMP. For example, I also run software which performs test DNS queries to Google, OpenDNS, and Cloudflare which is also showing many failed DNS queries per day.

The problem seems to affect the upload more than the download because speedtest.btwholesale.com indicates the download speed is 53Mbps which is normal but the upload speed is less than 1Mbps (it used to be around 12-15Mbps). speedtest.net shows similar speeds and also states around 18-23% packet loss. I am still able to download/stream quite reliably but anything involving the upload bandwidth is now incredibly slow and unreliable.

If you check the line stats, you should see drops at these dates and times; these were part of my troubleshooting:

Tues 29/03/2022 13:03, DrayTek router powered down
Tues 29/03/2022 13:34, DrayTek router powered up
Tues 29/03/2022 14:35, DrayTek router rebooted

Thurs 31/03/2022 10:57, DrayTek router unplugged
Thurs 31/03/2022 10:58, Plusnet router plugged in
Thurs 31/03/2022 11:06, Plusnet router unplugged; DrayTek router plugged in


In summary: There is a fault but I'm confident that the copper phone line is stable and working perfectly. I have performed the normal troubleshooting steps (test socket, different routers, disconnecting all LAN devices, etc) and also filled in the form on faults.plus.net over a week ago but heard nothing back.

I suspect the issue is somewhere between the DLSAM and the equipment in the exchange, do you have any tools which could check this part of the link?

Thank you.

[#2]

Thanks a lot for getting in touch  @hewitt and I'm really sorry to see that you're experiencing packet loss with your broadband connection, particularly on the upload.

I've just run some tests on the line from my side and can't find any issues that might be causing the packet loss to occur. 

Looking through the data you've provided, we can't see much wrong with the traceroute, the main thing is that it appears to have completed, I'd definitely recommend continuing to monitor this further via ThinkBroadband's Quality checker to see whether this continues, you'll find that here. This might shed some shed on issue.  

You won't be able to do this using our Plusnet hub One as it blocks ICMP/pings, but you should be able to use a third party one. 

If the problem continues over the next few days, we'd be open to raising this with Openreach, and arranging an engineer visit for further investigation.

[#3]

Hi,

Thanks for your response, @adam945

I signed up for the ThinkBroadband monitoring a few days after the issue started so I already have a few days worth of ping data in addition to my own monitoring data. Here is the live graph generated from the "graph share" function:

https://www.thinkbroadband.com/broadband/monitoring/quality/share/93713ba5b281e35051c2f7312a7a545407...

Unfortunately I forgot to update the page after an IP change so there is a gap in the monitoring recently.

I'll attach a graph from the 2nd April; all of the graphs that ThinkBroadband has generated so far look like this. The ThinkBroadband data matches my monitoring data; a constant loss of packets in the region of 10-20%.

I would also like to mention that I have been with Plusnet for over 2 years now and have never experienced anything like this, normally everything is very stable with no packet loss at all. Whatever this issue is, it came on very suddenly.

Thanks again.

EDIT: I forgot to mention, that latency spike at 1:30am ish is an automated offsite backup. Since the packet loss issue, any uploads are making the connection go very unstable (presumably a bufferbloat issue) as the upload speed is now in the kilobits.

[#4]

Hi,

I'm going to attach a screenshot from the DNS monitoring system I run. This performs a DNS query every minute to some of the major public DNS resolvers and plots the response time; timeouts are plotted as -128. This clearly shows it isn't just ICMP traffic being lost which may be expected over the public internet.

I will also include a screenshot from my MTR monitoring, this plot shows the packet response rate per hop. Although the MTR output in my initial post may look okay at first glance, this level of packet loss per host is very unusual and it started at the exact time I started experiencing packet loss to all hosts on the internet.

Both plots show a period before and after the problem began for comparison.

Thanks

[#5]

---

@adam945 wrote:

Looking through the data you've provided, we can't see much wrong with the traceroute,

I disagree. There's loss the minute the trace leaves @hewitt 's network.

Seems something is deifnitely amiss from the diagnostics that have been provided

[#6]

@bobpullen, that’s on me as I gave that advice.  Thanks for the heads up!

@hewitt, thanks for the detail. Coupled with the TBB quality monitor, unless Bob has other ideas, I think it’d be worth us arranging an Openreach engineer.

(Not in the office atm but we’ll pick this back up today)

[#7]

Thank you for the responses

I'm happy for you to arrange an engineer if you think that is the best way to proceed.

Just out of curiosity, does anybody know what hop 2 on my MTR is (172.16.18.15)? Since it has a private IP, I expect it to be some equipment before the packets reach the public internet but I'm curious whether it is in the cabinet or if it's equipment in the exchange.

Thanks again.

[#8]

Thanks a lot for getting back to us on this @hewitt, to be honest we wouldn't really know on our end, the second hop is likely a server at either the PCP or the Exchange. 

Are you still wanting to go ahead with an engineer visit? 

[#9]

Second hop is a Multi Service Edge Router (MSE).

[#10]

Hi @adam945, I am happy for you to arrange an engineer visit, thank you.

Thanks for the information about the second hop, @bobpullen.

I'll update you all once the engineer has done their thing.

[#11]

Hi @hewitt, can you drop us a reply back to the ticket I've opened on your account here please with your availability? https://www.plus.net/wizard/?p=view_question&id=223639573

[#12]

Thank you, @adamwalker, I have responded to the ticket with my availability.

[#13]

No problem at all! I've just made a booking for you and replied back to the ticket to confirm, please get back to me if there's any issues or anything else I can help with. - Adam

[#14]

The engineer was here this morning and tested the line, everything was normal and no faults were detected. They performed a DLM reset after their tests.

Our line now has interleaving enabled and is sycned slightly slower but I'm still observing the same level of packet loss so I'm confident this issue is caused by something further upstream rather than anything on our premises.

Since the line is now training again, I'm happy to just wait and monitor the situation until the line is stabilised. I'll post here again if there are any changes or if the problem is still here in 10 days or so.

Thanks for your help so far

[#15]

Thanks a lot for the update, yeah a slightly lower sync speed can be expected once interleaving has been turned on. Keep us posted, we'll be more than happy to follow up.