 |
Looks like the failure is limited to when the unbound resolver is locally installed and used. |
 |
We've had some intermittent reports of this but have not been able to nail down a root cause. How is your IPv6 traffic handled? Is it a direct allocation, or via a 646 tunnel broker like HE.net? Do you have any firewall rules on your network that would be preventing ICMP6 and/or PMTU traffic from flowing? |
|
|
It's an IPv6 only host in a datacentre (specifically Hetzner in DE). IPv6 works if the DNS points at a Mikrotik router. As soon as DNS is changed to point at a locally installed copy of the unbound nameserver, we fail. Looking at https://dnssec-analyzer.verisignlabs.com/mirrors.rockylinux.org it seems fastly fails DNSSEC checks. I wonder if resolvers are interpreting this as "DNS name not configured securely, pretend it does not exist". |
|
|
Thanks for the info, that's helpful. Let me reach out to Fastly and see if they have some thoughts here. DNSSEC is an ongoing project for us as an infrastructure team, too, but I'm not able to give an ETA at this time. |
|
|
In theory DNSSEC missing isn't an error, but if DNSSEC is misconfigured then resolution attempts will throw an error. Very odd that a CDN doesn't do DNSSEC. |
|
|
We have also been working on an ipv6 only network and had this problem on all of our rockylinux installations. We were running Nat64 with ipv6 on the inside and only ipv4 on the external network connection. By simply using the "-6" parameter with dnf, we succesfully updated without any further issues. I.E. "dnf -6 update" Hope this helps someone. |
|
|
The issue has reappeared. Last week it was still working. Now it's broken: Error: Failed to download metadata for repo 'devel': Cannot prepare internal mirrorlist: No URLs in mirrorlist |
|
|
ignore my last comment - this was supposed for a different issue report |
|
|
Some more info after this happened again. As per https://dnsviz.net/d/mirrors.rockylinux.org/dnssec/ the DNS name mirrors.rockylinux.org is hosted on DNS server that support both IPv6 and IPv4. The CNAME however points at Fastly, which for some bizarre reason has DNS that only supports IPv4 in 2025. The IPv6 only host cannot reach the IPv4 only DNS, and so we get "Could not resolve host". Fastly needs to fix their DNS. |
|
|
I recall this being an issue in CentOS 6 and/or 7. The preferred addressing is IPv6 when it's enabled. Is there something in the path that isn't IPv6-enabled, which is dropping the requests before they reach Fastly. If you run ''mtr -6 mirrors.rockylinux.org'', it should show when the break is happening. If the break is happening with Unbound locally, I'd love to see how it's configured. I notice that Fastly has IPv6 addresses when querying public resolvers. I'm quite curious about what you see in ''mtr'' and how DNS is configured in your environment. Can you share any of that info? |
|
|
Look closely at the very last entry on the bottom right of https://dnsviz.net/d/mirrors.rockylinux.org/dnssec/ - you see the name rockylinux.map.fastly.net has an AAAA record, but this AAAA record is only present on a series of four DNS servers, all four of which have IPv4 addresses, and zero of which have IPv6 addresses like everything else. An IPv6-only name server cannot contact the IPv4-only name servers of rockylinux.map.fastly.net, and so the lookup returns SERVFAIL. Lots of people don't see this because while the host might be IPv6-only, and the network might be IPv6 only, if the DNS is dual stack then the IPv4 addresses are reachable and the request "works". If you have a pure IPv6 host, a pure IPv6 network, AND a pure IPv6 DNS server, you get SERVFAIL. |
|
|
One further detail - has Rockylinux followed the instructions here: https://www.fastly.com/documentation/guides/full-site-delivery/domains-and-origins/enabling-dualstack-connections/ "Enabling dualstack on customer-specific hostnames If you have a customer-specific hostname, contact support and we'll provide you with a parallel IPv6 map or enable dualstack on your current one." |
|
|
This is still broken on rocky10. The mirrors.rockylinux.org DNS name is CNAME'd to a DNS name that is hosted exclusively on IPv4, and so always fails on an IPv6 host. |
|
|
> The mirrors.rockylinux.org DNS name is CNAME'd to a DNS name that is hosted exclusively on IPv4, and so always fails on an IPv6 host. No it's not, not has it ever. ``` ╰─⊙ host mirrors.rockylinux.org 130 ↵ mirrors.rockylinux.org is an alias for dualstack.dl.map.rockylinux.org. dualstack.dl.map.rockylinux.org is an alias for rockylinux.map.fastly.net. rockylinux.map.fastly.net has address 199.232.198.132 rockylinux.map.fastly.net has address 199.232.194.132 rockylinux.map.fastly.net has IPv6 address 2a04:4e42:4d::644 rockylinux.map.fastly.net has IPv6 address 2a04:4e42:4c::644 ``` |
|
|
Furthermore: ``` ╰─➤ curl -6 https://mirrors.rockylinux.org/mirrorlist # either path=, or repo= and arch= must be specified ``` Let's not confuse issues here. IPv6 works perfectly fine. Strict DNSSEC is the issue you're running up against. |
|
|
For the avoidance of doubt, I'm not talking about the mirror address, I am rather talking about the DNS name rockylinux.map.fastly.net which is hosted on IPv4 only - that means only IPv4 name servers - even though the resolved name points at an IPv6 address. If you can't reach the name server (because the name server is IPv4 only), you can't resolve the AAAA record, even if the AAAA record is present. The dnsviz site draws the problem on a diagram, attached below. https://dnsviz.net/d/mirrors.rockylinux.org/dnssec/ The red oval on the right (the AAAA record) is hosted exclusively on the list of IPv4 nameservers on the red oval on the left (the servers). If you host an IPv6 AAAA record on an IPv4 name server, it won't work. For an example of how it should be configured, the mirrors.rockylinux.org address is correctly hosted by both IPv4 and IPv6 name servers, and thus is accessible to both IPv4 and IPv6 hosts. You're hosting your own DNS correctly, fastly is not. |
|
|
This is an IPv6 only host, querying an IPv6 only name server: [root@robinhood ~]# drill -6 mirrors.rockylinux.org AAAA ;; ->>HEADER<<- opcode: QUERY, rcode: SERVFAIL, id: 38988 ;; flags: qr rd ra ; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;; mirrors.rockylinux.org. IN AAAA ;; ANSWER SECTION: ;; AUTHORITY SECTION: ;; ADDITIONAL SECTION: ;; Query time: 1236 msec ;; SERVER: ::1 ;; WHEN: Wed Dec 10 16:26:06 2025 ;; MSG SIZE rcvd: 40 |
|
|
> If you host an IPv6 AAAA record on an IPv4 name server, it won't work. This is incorrect. AAAA records served by IPv4-only authoritative nameservers work fine--this is standard DNS behavior. The issue you're experiencing is that your recursive resolver is IPv6-only and cannot contact our IPv4-only authoritative nameservers. The solution is to configure a dual-stack recursive resolver or use NAT64/DNS64. This is an edge-case resolver configuration issue affecting users running IPv6-only recursive resolvers without NAT64/DNS64--a deployment scenario that represents a very small fraction of our user base, and one that has straightforward resolver-side solutions. That said, I did reach out to Fastly last year to be added to their beta program for dualstack authoritative DNS, and we _do_ have a dualstack map we can attempt to use. I have been reluctant to make any change in-situ due to the small gain for some users compared to the risk of affecting all the other traffic we serve on a daily basis--particularly when there are fairly easy workarounds on the end of the user encountering such an issue, namely: using a dual stack resolver capable of reaching Fastly's existing DNS infrastructure, hardcoding a list of mirrors in the repofile, or several other options. I am still reluctant to make this change in the current service, but as we have been working on plans to replace the mirrormanager infrastructure entirely with new systems and CDN config, I am intending to incorporate this change into that project. Note this will only affect mirrors.rockylinux.org and not dl/download(s).rockylinux.org. In the interim, I can look at creating a new CNAME to this map on an alternative URL so you can test it on your infrastructure to ensure it works as we hope it will--though I can't make promises on when that might be live. --Neil |
- Anonymous
