IPv6 Connectivity Problems

[#1]

Hi Support/Everyone,
From Mon 2nd July I seem to have lost IPv6 connectivity and whilst I'm pretty sure I didn't change anything that day (I was at work when the failure occured (10-11am)) I cannot rule out something like a DHCP expiry unearthing a latent configuration error at my end.
I am using a Cisco 877 router and can obtain an IPv6 prefix via DHCP:

Cisco877#clear ipv6 dhcp client dial 1
Cisco877#sh ipv6 dhcp int dial 1
Dialer1 is in client mode
  Prefix State is OPEN (0)
  Information refresh timer expires in 23:59:54
  Renew will be sent in 7w0d
  Address State is IDLE
  List of known servers:
    Reachable via address: FE80::90:1A00:3A3:8D54
    DUID: 000200000A4C453332302F373435414333334558322F03
    Preference: 0
    Configuration parameters:
      IA PD: IA ID 0x000D0001, T1 INFINITY, T2 INFINITY
        Prefix: 2A02:16C8:2000:600::/56
                preferred lifetime INFINITY, valid lifetime INFINITY
      DNS server: 2001:470:20::2
      Information refresh time: 0
  Prefix name: MYPLUSNETPREFIX
  Prefix Rapid-Commit: disabled
  Address Rapid-Commit: disabled

The router then sets itself a link-local IPv6 address and a global IPv6 address on VLAN1 with this prefix using auto-configuration:

Vlan1                      [up/up]
    FE80::B6A4:E3FF:FE11:A57A
    2A02:16C8:2000:600:B6A4:E3FF:FE11:A57A

At this point I can ping the DHCP server using (from) my link local address:

Cisco877#ping ipv6 FE80::90:1A00:3A3:8D54
Output Interface: dial 1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to FE80::90:1A00:3A3:8D54, timeout is 2 seconds:
Packet sent with a source address of FE80::B6A4:E3FF:FE11:A57A
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 36/39/40 ms

But I cannot ping anything beyond that (e.g. the DNS server referenced in the DHCP response) from my global address:

Cisco877#ping ipv6 2001:470:20::2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 2001:470:20::2, timeout is 2 seconds:
HHHHH
Success rate is 0 percent (0/5)

Similarly, I cannot ping in from an external IPv6-enabled machine:

mathew@targur:~$ ping6 2A02:16C8:2000:600:B6A4:E3FF:FE11:A57A
PING 2A02:16C8:2000:600:B6A4:E3FF:FE11:A57A(2a02:16c8:2000:600:b6a4:e3ff:fe11:a57a) 56 data bytes
From 2001:ba8:0:1f1::1 icmp_seq=1 Destination unreachable: No route
From 2001:ba8:0:1f1::1 icmp_seq=2 Destination unreachable: No route
From 2001:ba8:0:1f1::1 icmp_seq=3 Destination unreachable: No route
^C
--- 2A02:16C8:2000:600:B6A4:E3FF:FE11:A57A ping statistics ---
3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 2006ms

The 'no route' response is interesting, and if we look at the traceroute it doesn't seem to be getting even as far as Plusnet, nevermind all the way to me:

mathew@targur:~$ traceroute6 2A02:16C8:2000:600:B6A4:E3FF:FE11:A57A
traceroute to 2A02:16C8:2000:600:B6A4:E3FF:FE11:A57A (2a02:16c8:2000:600:b6a4:e3ff:fe11:a57a), 30 hops max, 40 byte packets
 1  2001:ba8:1f1:f137::1 (2001:ba8:1f1:f137::1)  0.151 ms  0.194 ms  0.181 ms
 2  jump-gw-1.ipv6.bitfolk.com (2001:ba8:0:1f1::1)  0.906 ms !N  0.990 ms !N  1.367 ms !N

Are you able to check/confirm that my prefix is actually being announced via BGP?
Thanks for reading this far, and if there's anything anyone can suggest it'd be appreciated. It's always frustrating when it has been working for a long time (over a year now) and then one day just stops!! That said, the short traceroute suggests it's a Plusnet issue but I stand to be corrected.
Mathew

[#2]

I have checked on two connections and both have IPv6 connection problems.
Both are on the trial and have been working OK until recently.
PeteB

[#3]

thinkbroadband.com quality monitor is showing 100% packet loss for ipv6, ipv4 is OK.
I suspect some change has been made by Plusnet that has caused the ipv6 connectivity to stop working.
PeteB

[#4]

Thanks Pete. That's quite reassuring in a perhaps-its-not-something-I've-done kind of way!
Mathew

[#5]

One of the routers is a 7800N and the other a TG582n so the problem is not related to the hardware being used.
Last ipv6 connection on the quality monitor is approx. 10:30 on Monday 2nd July 2012.
Pete

[#6]

That coincides with when mine went down so thanks for mentioning that.
Mathew

[#7]

Incidentally, the consequences of this situation - apparent IPv6 capability but no actual offsite connectivity - has caused problems for some devices which might be worthy of consideration for an eventual future rollout.
Firstly, there is the relatively common symptom of browsers (Firefox 13 here) attempting to connect over IPv6 and falling back to IPv4 only after a timeout thus causing sluggish browsing to sites with IPv6 AAAA DNS records. The 'Happy Eyeballs'[1] technique (and similar) may eventually alleviate this problem but for now it is rather frustrating.
Worse than sluggish performance though I am finding that some applications behave very badly. For example, Google's 'Play' (app store) on Android seems to trip over itself if it thinks it has IPv6 connectivity yet fails to get packets delivered - it just waits and waits before eventually giving up (i.e. it doesn't fall back to IPv4). This is occuring on two devices, both running Gingerbread 2.3.5.
This problem is likely to get far worse before it starts to get better as more and more sites are starting to add AAAA records now that they're supporting IPv6.
Mathew
[1] It would appear that Firefox does implement HE but it doesn't seem to be working particularly effectively. To be fair the delay is in the order of a few seconds, and so a lot better than the 10's of seconds that used to occur a few years ago with a broken IPv6 setup, but in today's want-it-now society I'm sure it'll still be too long for many people!

[#8]

Similar problems here.
I managed to ping your IPv6 address above, MJN, though the ping time was >600ms.
I have disabled IPv6 on my router for now as it's currently a pain.

[#9]

Hi Support,

Quote from: MJN

Are you able to check/confirm that my prefix is actually being announced via BGP?

Just to follow up on this, your entire prefix has dropped out of the routing table:

route-views.linx.routeviews.org> show ipv6 bgp 2a02:16c8::/32
% Network not in table

Mathew

[#10]

Morning folks.
I am doing some digging to see if we can see what may be going on. There are some minor suspicions giving some of the recent network maintenance we have been undertaking, but they are only that initially.
Can I confirm though, does IPv4 connectivity remain unaffected when using the IPv6 test account?

[#11]

Yes, can confirm that IPv4 is fine on the test account.
It looks to me like it's simply that your BGP announcements to the outside world for IPv6 have stopped. I suppose though that that could be a consequence of something gone awry internally so more of a symptom than the cause?
Mathew

[#12]

IPv4 also OK on both my and my friend's IPv6 test accounts.
Pete

[#13]

MJN,
Our original thoughts on a cause may not be to blame, though until I get a response from our network operations team, we are not ruling out any fringe side-effects.
PeteB,
I cannot locate your main account, thus cannot workout which IPv6 test account you are on.
Can you let me know the main username of the line you are testing IPv6 on, as I can get everything else from this.
Until I have this, I cannot work the rest out.

[#14]

PM sent with account information.
PeteB

[#15]

Thanks.
Checking through that, I thought was had seen a common denominator of you being on PCL gateways. However, the second of the two accounts PeteB quoted in presently on PTN.
One of our newer Network Ops has had a look (I say newer, he's actually a returning member), who is not fully upto speed on the setup that is in for our IPv6 trials.
He has confirms that part of our core does not do IPv6 routing as of yet, but that could ultimately be per present design. His reply also indicates he thinks something is wrong our side, but all elements on balance suggests we could also be looking in the wrong places.
I'm waiting to find out if one of our other core network engineers is around who most certainly knows the IPv6 elements.