I'm trying to get to the bottom of a possible routing issue where I can't seem to connect through to our local hospitals NHS website. I've noticed this has been an issue a few times before, and I thought nothing of it until recently when I was trying to get some specific information.
The website in question is https://www.boltonft.nhs.uk/
Any attempt to connect to it (curl, chrome, any other browser) simply results in a timeout. Here it the output from curl and traceroute when ran within my network:
root@raspi01 ~ # traceroute -I www.boltonft.nhs.uk
traceroute to www.boltonft.nhs.uk (213.104.98.149), 30 hops max, 60 byte packets
1 254.core.plus.net (195.166.130.254) 13.194 ms 13.203 ms 13.177 ms
2 84.93.253.115 (84.93.253.115) 13.468 ms 13.700 ms 13.707 ms
3 core1-BE1.southbank.ukcore.bt.net (195.99.125.130) 13.151 ms 13.191 ms 13.664 ms
4 peer3-et-0-0-2.redbus.ukcore.bt.net (62.172.103.240) 13.754 ms 13.777 ms 13.781 ms
5 109.159.253.63 (109.159.253.63) 20.649 ms 20.696 ms 20.733 ms
6 * * *
7 bagu-core-2a-ae10-0.network.virginmedia.net (62.254.84.2) 25.504 ms 23.142 ms 23.385 ms
8 bagu-metnet-3b-lag-56.network.virginmedia.net (82.8.124.178) 23.168 ms 23.247 ms 23.272 ms
9 bnft-bl4-ia1.network.virginmedia.net (213.104.213.126) 24.374 ms 24.743 ms 24.755 ms
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *
root@raspi01 ~ # curl -v -m30 https://www.boltonft.nhs.uk/ -o out
* Expire in 0 ms for 6 (transfer 0x671950)
* Expire in 30000 ms for 8 (transfer 0x671950)
* Expire in 1 ms for 1 (transfer 0x671950)
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Expire in 1 ms for 1 (transfer 0x671950)
* Expire in 2 ms for 1 (transfer 0x671950)
* Expire in 1 ms for 1 (transfer 0x671950)
* Expire in 1 ms for 1 (transfer 0x671950)
* Expire in 1 ms for 1 (transfer 0x671950)
* Trying 213.104.98.149...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x671950)
0 0 0 0 0 0 0 0 --:--:-- 0:00:29 --:--:-- 0* Connection timed out after 30001 milliseconds
0 0 0 0 0 0 0 0 --:--:-- 0:00:30 --:--:-- 0
* Closing connection 0
curl: (28) Connection timed out after 30001 milliseconds
If I access this via a different connection (in this case route out via VPN, then the site returns just fine with no issues:
root@raspi01 ~ # traceroute -T www.boltonft.nhs.uk
traceroute to www.boltonft.nhs.uk (213.104.98.149), 30 hops max, 60 byte packets
1 10.35.0.1 (10.35.0.1) 17.397 ms 17.383 ms 17.502 ms
2 te-3-3-4006.pe3.man4.uk.m247.com (217.64.114.161) 18.457 ms 19.779 ms 19.908 ms
3 xe-1-2-1-0.core1.man4.uk.m247.com (83.97.21.144) 18.941 ms vlan2902.bb1.fra2.de.m247.com (82.102.29.128) 20.146 ms xe-2-1-0-0.core1.man4.uk.m247.com (77.243.185.12) 20.035 ms
4 te-12-3-0.core-dc2.man4.uk.m247.com (83.97.21.70) 47.011 ms te-13-4-0.core-dc2.man4.uk.m247.com (77.243.176.47) 47.090 ms te-12-3-0.core-dc2.man4.uk.m247.com (83.97.21.70) 46.992 ms
5 te-5-8-0.bb1.man2.uk.m247.com (77.243.185.137) 20.013 ms 20.061 ms te-6-5-0.bb1.man2.uk.m247.com (77.243.185.1) 20.102 ms
6 tcma-ic-2-xe-210-0-0.network.virginmedia.net (212.250.14.189) 20.118 ms 18.890 ms 18.910 ms
7 * * *
8 bagu-core-2a-ae10-0.network.virginmedia.net (62.254.84.2) 20.681 ms 20.534 ms 20.497 ms
9 bagu-metnet-3b-lag-56.network.virginmedia.net (82.8.124.178) 20.556 ms 20.569 ms 20.592 ms
10 bnft-bl4-ia1.network.virginmedia.net (213.104.213.126) 20.860 ms 20.891 ms 20.795 ms
11 91-187-250-212.static.virginm.net (212.250.187.91) 20.805 ms 25.048 ms 25.142 ms
12 149.98-104-213.static.virginmediabusiness.co.uk (213.104.98.149) 24.953 ms 25.062 ms 25.025 ms
root@raspi01 ~ # curl -v -m30 https://www.boltonft.nhs.uk/ -o out
* Expire in 0 ms for 6 (transfer 0xa74950)
* Expire in 30000 ms for 8 (transfer 0xa74950)
* Expire in 1 ms for 1 (transfer 0xa74950)
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Expire in 1 ms for 1 (transfer 0xa74950)
* Expire in 2 ms for 1 (transfer 0xa74950)
* Expire in 1 ms for 1 (transfer 0xa74950)
* Expire in 1 ms for 1 (transfer 0xa74950)
* Expire in 1 ms for 1 (transfer 0xa74950)
* Trying 213.104.98.149...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0xa74950)
* Connected to www.boltonft.nhs.uk (213.104.98.149) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [21 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [3030 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
* subject: C=GB; L=Bolton; O=Bolton NHS Foundation Trust; CN=*.boltonft.nhs.uk
* start date: Jan 7 00:00:00 2022 GMT
* expire date: Jan 5 23:59:59 2023 GMT
* subjectAltName: host "www.boltonft.nhs.uk" matched cert's "*.boltonft.nhs.uk"
* issuer: C=US; O=DigiCert Inc; CN=DigiCert TLS RSA SHA256 2020 CA1
* SSL certificate verify ok.
} [5 bytes data]
> GET / HTTP/1.1
> Host: www.boltonft.nhs.uk
> User-Agent: curl/7.64.0
> Accept: */*
>
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [265 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [265 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
< HTTP/1.1 200 OK
< Date: Thu, 28 Jul 2022 12:39:49 GMT
< Server: Apache/2.4.41 (Ubuntu)
< Link: <https://www.boltonft.nhs.uk/>; rel=shortlink
< Vary: Accept-Encoding
< Transfer-Encoding: chunked
< Content-Type: text/html; charset=UTF-8
<
{ [6 bytes data]
100 278k 0 278k 0 0 721k 0 --:--:-- --:--:-- --:--:-- 723k
* Connection #0 to host www.boltonft.nhs.uk left intact
Weirdly, I have to perform ICMP traceroute when running through Plusnet otherwise the TCP one used earlier returns the same hop multiple times:
root@raspi01 ~ # traceroute -T www.boltonft.nhs.uk
traceroute to www.boltonft.nhs.uk (213.104.98.149), 30 hops max, 60 byte packets
1 149.98-104-213.static.virginmediabusiness.co.uk (213.104.98.149) 10.311 ms 10.278 ms 10.556 ms
2 149.98-104-213.static.virginmediabusiness.co.uk (213.104.98.149) 11.169 ms 11.113 ms 11.372 ms
3 149.98-104-213.static.virginmediabusiness.co.uk (213.104.98.149) 11.366 ms 11.282 ms 10.925 ms
4 149.98-104-213.static.virginmediabusiness.co.uk (213.104.98.149) 11.473 ms 11.168 ms 11.137 ms
5 149.98-104-213.static.virginmediabusiness.co.uk (213.104.98.149) 20.637 ms 18.463 ms 18.378 ms
6 * * *
7 149.98-104-213.static.virginmediabusiness.co.uk (213.104.98.149) 28.309 ms * 28.130 ms
8 149.98-104-213.static.virginmediabusiness.co.uk (213.104.98.149) 28.190 ms 28.067 ms 28.014 ms
9 149.98-104-213.static.virginmediabusiness.co.uk (213.104.98.149) 28.021 ms 27.969 ms 27.915 ms
10 149.98-104-213.static.virginmediabusiness.co.uk (213.104.98.149) 27.915 ms 27.904 ms 27.809 ms
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *
So yes, a bit of a loss but also puzzled at the odd behaviour of running traceroute over tcp when going out via Plusnet. This behaviour appears to happen to every endpoint via tcp.
I've also tethered to my mobile phones data connection and ran the same tests, and they worked just fine including showing the correct hops via traceroute when going via tcp.
For reference, I run pfSense which has multiple gateways configured:
WAN -> Plusnet
OVPN -> Via external VPN Provider
HEV -> Hurricane Electric IPv6 Gateway (not used for this test)
To test the connection, I change the outbound firewall rule to send traffic via the OpenVPN gateway instead of the Plusnet gateway for a particular destination (in this case the IP in which resolves against the Bolton NFT Trust website).
