 |
Hello, I am not able to reproduce this from my location, nor from several proxy locations around the globe. Can you let me know where you are querying from, so I can try to isolate this to a cache region? I checked all our servers, and they are still all sharing the same root NFS share which provides the content in the proper location. As a point of procedure, the Rocky Enterprise Software Foundation does not provide support for EL8.6, only the latest release of each active Major. Best, Neil |
 |
The ISP is located in Albuquerque, New Mexico. |
|
|
Is there a better URL that we can use for refreshing our own local mirror? |
|
|
Yesterday afternoon, we found bad metadata in they rocky 9 and rocky 9.2 both in the supported and vault directory trees. After cleaning out the cache and getting good metadata, the Rocky 9.2 got bad metadata again from updating the cache. This morning we found bad metadata in the rocky 8, 8.5, 8.6, and 8.7 cached metadata. We are seeing versions that should only be in the vault directory tree showing up in the supported directory tree. We are seeing supported versions that are also in the vault directory tree where the vault cached copy has bad metadata. |
 |
Hello. Can you provide answers to the following: * What is the exact mirror you are having issues with? You noted in the original post that this is a "local mirror cache". Since this is the case, we are assuming that this is an internal mirror to you. How are you syncing the repo contents? rsync or reposync? * Can you define "bad metadata"? What is the behavior you're actually seeing? * If this is *not* a local mirror that you are syncing and is dl.rockylinux.org, can you please provide the IP's that you are actually hitting? It would be help us to know what part of the CDN you are hitting. As you know, we do not provide older versions in /pub/rocky. Majority, if not all of our mirrors use rsync to sync from the vault or pub. There are also no reports of issues with our repos from dl.rockylinux.org or our rsync endpoint, so it will be helpful to know more information to try to resolve your issue. |
|
|
We are using a product called Jfrog Artifactory to cache access to both http://dl.rockylinux.org/pub/rocky/ and http://dl.rockylinux.org/vault/rocky/ as two remote repositories. We have created an Artifactory virtual repository that includes both of of the above remote repositories. The way Artifactory works is that when a request for a file is made, it checks its local cache and if that cache entry is not expired, it uses that entry. If the cache entry is expired, checks to see if there is a new copy to fetch. The virtual repository looks first for a file in the /pub/rocky/ remote repository and if it is not found, it looks the /vault/rocky remote repository. We are using the Artifactory default period of time of 2 hours before checking for updated metadata. We are seeing directory trees show up periodically in /pub/rocky cache that should only be in /vault/rocky that are apparently older than the current contents of /vault/rocky. When looking at your /pub/rocky with a web browser, we do not see those directory trees. We delete those trees from the cache, yet sometimes they show back up again, in a few cases on the same day. We are also seeing directory trees showing up in the /vault/rocky cache that that appear to be older copies of files in the /pub/rocky directory tree. When looking at your /vault/rocky with a web browser, we do not see those directory trees. We delete those directory trees from the cache, yet sometimes they show back up again. This is not every day, it just seems to happen periodically every few weeks. Artifactory is only configured with those above two URLs to fetch files for caching. Typical error seen: dnf makecache --releasever=9.2 rocky9-base-artifactory 4.1 MB/s | 1.7 MB 00:00 Errors during downloading metadata for repository 'rocky9-base-artifactory': - Downloading successful, but checksum doesn't match. Calculated: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855(sha256) Expected: 10d873367f6e6584ffa5248b2154cf820df4d9b408de63933e2feb8b5073a408(sha256) Error: Failed to download metadata for repo 'rocky9-base-artifactory': Yum repo downloading error: Downloading error(s): repodata/a995b6bd-aaea-4b98-b18c-ed756f9bf243-GROUPS.xml.gz - Cannot download, all mirrors were already tried without success; repodata/30cdb1ea-9f14-41a9-8f71-15744f155975-UPDATEINFO.xml.gz - Cannot download, all mirrors were already tried without success Artifactory logs its outgoing fetches in artifactory-request-out.log. Unfortunately the log does not contain any IP addresses involved in the fetching of the files. I have seen other tickets of people reporting similar behavior that were closed because they were referencing rocky versions that were no longer in support, so we are not the only site that is seeing this. In addition, sometimes the dl.rockylinux.org load balancer is returning a file with containing an HTTP error code instead of the expected file, as has been reported on a different ticket. If there is a better IP or URL that we can use to make sure that we are always getting a known good directory we can change Artifactory to cache from that. |
|
|
If I'm understanding correctly, you are using a tool to "cache" the contents of our repositories, instead of syncing them locally. >We are seeing directory trees show up periodically in /pub/rocky cache that should only be in /vault/rocky that are apparently older than the current contents of /vault/rocky. >When looking at your /pub/rocky with a web browser, we do not see those directory trees. >We delete those trees from the cache, yet sometimes they show back up again, in a few cases on the same day. >We are also seeing directory trees showing up in the /vault/rocky cache that that appear to be older copies of files in the /pub/rocky directory tree. >When looking at your /vault/rocky with a web browser, we do not see those directory trees. >We delete those directory trees from the cache, yet sometimes they show back up again. This is beginning to sound like is you have cache pollution that needs to be resolved within artifactory. There is typically no reason for older content to ever show up in the CDN nor through our LB (assuming bypass of the CDN). But... >Artifactory logs its outgoing fetches in artifactory-request-out.log. Unfortunately the log does not contain any IP addresses involved in the fetching of the files. Unfortunately without any IP's, we won't be able to investigate if it's truly our CDN or our LB, to rule out cache pollution in your caching system. So it will not be clear where the issue originates from. >I have seen other tickets of people reporting similar behavior that were closed because they were referencing rocky versions that were no longer in support, so we are not the only site that is seeing this. Can you please reference these other reports? >In addition, sometimes the dl.rockylinux.org load balancer is returning a file with containing an HTTP error code instead of the expected file, as has been reported on a different ticket. It would be helpful to have historic logs that show how many times this has happened to you. We very rarely receive reports about this and when we do, it's because we are doing maintenance on our infrastructure or resolving an outage, as we did yesterday evening GMT-7. "Sometimes" isn't enough for us to go off of. >If there is a better IP or URL that we can use to make sure that we are always getting a known good directory we can change Artifactory to cache from that. I should note that dl.rockylinux.org is our tier 0 and does not redirect elsewhere to another mirror (it's at our CDN and ultimately leads to our tier 0 infra in us east 2 in AWS). Setting ticket to needinfo. |
|
|
Would that outage yesterday evening been around 8:00 pm GMT? The problems for the most recent incidents was first noticed on Oct 5, at 3:41 pm US Central time, or 8:41 pm GMT and the last report I have has a timestamp of 12:48 GMT on October 6. The end time is not exact as I had to manually clear the cache to fix the condition. Timestamps for previous incidents: [2023-09-01T02:07:58.837Z] Error: Failed to download metadata for repo '-rocky8-powertools-artifactory': repomd.xml parser error: Parse error at line: 1 (Extra content at the end of the document.. This resulted in https://bugs.rockylinux.org/view.php?id=4126 being filed. [2023-09-27T10:44:36.092Z] Error: Failed to download metadata for repo 'rocky8-powertools-artifactory': repomd.xml parser error: Element <repomd> was not found - Bad repomd file And then yesterday: [2023-10-06T12:47:54.934Z] Errors during downloading metadata for repository 'rocky9-base-artifactory': [2023-10-06T12:47:54.934Z] - Downloading successful, but checksum doesn't match. Calculated: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855(sha256) Expected: 10d873367f6e6584ffa5248b2154cf820df4d9b408de63933e2feb8b5073a408(sha256) [2023-10-06T12:48:00.130Z] rocky8-powertools-artifactory 0.0 B/s | 0 B 00:00 [2023-10-06T12:48:00.130Z] Error: Failed to download metadata for repo 'daos_ci-rocky8-powertools-artifactory': repomd.xml parser error: Parse error at line: 1 (Extra content at the end of the document [2023-10-06T12:48:00.130Z] ) In a current search of the bug tracker, I can not find the tickets that I found earlier that were closed for being for an older version. |
|
|
@John Malmberg > Yesterday afternoon, we found bad metadata in they rocky 9 and rocky 9.2 both in the supported and vault directory trees. > After cleaning out the cache and getting good metadata, the Rocky 9.2 got bad metadata again from updating the cache. > > This morning we found bad metadata in the rocky 8, 8.5, 8.6, and 8.7 cached metadata. > > We are seeing versions that should only be in the vault directory tree showing up in the supported directory tree. > We are seeing supported versions that are also in the vault directory tree where the vault cached copy has bad metadata. This is different from what was originally reported in this ticket. What you are reporting above is being addressed in ticket 0004126. Please let's not conflate these two issues. |
|
|
Hello. This is a notification that this bug will be closed. We are closing this ticket due to the following: * Rocky Linux 9.3 has been released, 9.2 is vaulted. 8.9 soon to follow with 8.8 to be vaulted. * We have not received any reports from end users of issues from the CDN, mirror manager, or our mirrors * Ticket has gone off-topic from the original report, conflating an issue from the reported ticket 0004126 * The issue appears to be a problem either with the application artifactory or your outbound proxy mechanisms as noted in 0004126 If you are still experiencing the issues as in the original stated report, please open a new bug report with previously presented data, as well as additional information such as application logs and outbound proxy logs. After opening the report, you can set this ticket as related in the "relationships" box below the report after submission. Thank you. We hope you enjoy your holiday. |
- Anonymous
