 |
I just wanted to add my voice to this issue. We are too using pulp for rocky linux repo mirroring and encounter the same issue. This is really impacting us (no new server install for now) as we don't use versionning feature on pulp. Maybe minor is not the right severity.... Thanks |
 |
I would argue that the error being reported from pulp is incorrect. It has nothing to do with "out of order" metadata and more so the metadata itself being inconsistent. Between two pieces of the metadata, there are shared bits such as checksums. Unfortunately, you did not provide the error output you were seeing. But it is likely you were seeing two sha sums that were stated as "out of order" when instead the problem is that the sums are not matching between the primaries and file lists. This should be fixed and a sync should work normally. If you are syncing from a mirror, please give them time to sync. If you are syncing directly from dl.rl.o, it should work right away, assuming fastly cache has been purged in your respective pops. |
|
|
Message coming from pulp is "Msg: Parsing interrupted: Out of order metadata" I sync every hour from https://ftp.gwdg.de/pub/linux/rocky/9.1/BaseOS/x86_64/os/ The error happens from "Jan 27 07:00:00" until "Jan 31 08:00:00" Means the last sync at 09:00:00 was successful |
|
|
fullstack trace from pulp is --- pulp [579b6c3ff3a445f39715b8a9b722e74c]: pulpcore.tasking.pulpcore_worker:INFO: File "/usr/local/lib/python3.8/site-packages/pulpcore/tasking/pulpcore_worker.py", line 452, in _perform_task result = func(*args, **kwargs) File "/usr/local/lib/python3.8/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 565, in synchronize repo_version = dv.create() or repo.latest_version() File "/usr/local/lib/python3.8/site-packages/pulpcore/plugin/stages/declarative_version.py", line 161, in create loop.run_until_complete(pipeline) File "/usr/lib64/python3.8/asyncio/base_events.py", line 616, in run_until_complete return future.result() File "/usr/local/lib/python3.8/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline await asyncio.gather(*futures) File "/usr/local/lib/python3.8/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__ await self.run() File "/usr/local/lib/python3.8/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 856, in run await self.parse_repository_metadata(repomd, repomd_files) File "/usr/local/lib/python3.8/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 908, in parse_repository_metadata await self.parse_packages( File "/usr/local/lib/python3.8/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 1281, in parse_packages for pkg in parser.as_iterator(): |
|
|
I missed the first line from the stack trace pulp [579b6c3ff3a445f39715b8a9b722e74c]: pulpcore.tasking.pulpcore_worker:INFO: Task 4f5a132d-1624-42e0-8d5f-1f6163b1c622 failed (Parsing interrupted: Out of order metadata: d2134ac3dc9c0b26098b6e5d50a51f2a47df3bd8b4801369f39a815c0cfdba10 vs cf36b2caa37bf0e45b7c3d456a4897c6ab645367e5286c28bcac77b92d6c9d04. |
|
|
Here is our pulp output ``` an 30 13:29:53 pulp01.it.pasteur.fr pulpcore-worker[3001341]: pulp [d104d89f04454a76acbc94bbdf02e535]: pulpcore.tasking.pulpcore_worker:INFO: Task b94008d7-f7d3-4ac4-91cd-17169f4ede02 failed (Parsing interrupted: Out of order metadata: 786010ce275e9a122265d52515eb90784e036bdf68f2ed2d78d1de250016d3f0 vs be8f8f34a444f82e2d057d799e1eaf4c69197efeb1a46ecd2d4cc1a7ea37949f.) Jan 30 13:29:53 pulp01.it.pasteur.fr pulpcore-worker[3001341]: pulp [d104d89f04454a76acbc94bbdf02e535]: pulpcore.tasking.pulpcore_worker:INFO: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulpcore/tasking/pulpcore_worker.py", line 452, in _perform_task Jan 30 13:29:53 pulp01.it.pasteur.fr pulpcore-worker[3001341]: result = func(*args, **kwargs) Jan 30 13:29:53 pulp01.it.pasteur.fr pulpcore-worker[3001341]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 565, in synchronize Jan 30 13:29:53 pulp01.it.pasteur.fr pulpcore-worker[3001341]: repo_version = dv.create() or repo.latest_version() Jan 30 13:29:53 pulp01.it.pasteur.fr pulpcore-worker[3001341]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulpcore/plugin/stages/declarative_version.py", line 161, in create Jan 30 13:29:53 pulp01.it.pasteur.fr pulpcore-worker[3001341]: loop.run_until_complete(pipeline) Jan 30 13:29:53 pulp01.it.pasteur.fr pulpcore-worker[3001341]: File "/usr/lib64/python3.9/asyncio/base_events.py", line 642, in run_until_complete Jan 30 13:29:53 pulp01.it.pasteur.fr pulpcore-worker[3001341]: return future.result() Jan 30 13:29:53 pulp01.it.pasteur.fr pulpcore-worker[3001341]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline Jan 30 13:29:53 pulp01.it.pasteur.fr pulpcore-worker[3001341]: await asyncio.gather(*futures) Jan 30 13:29:53 pulp01.it.pasteur.fr pulpcore-worker[3001341]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__ Jan 30 13:29:53 pulp01.it.pasteur.fr pulpcore-worker[3001341]: await self.run() Jan 30 13:29:53 pulp01.it.pasteur.fr pulpcore-worker[3001341]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 856, in run Jan 30 13:29:53 pulp01.it.pasteur.fr pulpcore-worker[3001341]: await self.parse_repository_metadata(repomd, repomd_files) Jan 30 13:29:53 pulp01.it.pasteur.fr pulpcore-worker[3001341]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 908, in parse_repository_metadata Jan 30 13:29:53 pulp01.it.pasteur.fr pulpcore-worker[3001341]: await self.parse_packages( Jan 30 13:29:53 pulp01.it.pasteur.fr pulpcore-worker[3001341]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 1281, in parse_packages Jan 30 13:29:53 pulp01.it.pasteur.fr pulpcore-worker[3001341]: for pkg in parser.as_iterator(): ``` |
|
|
my sentence "Means the last sync at 09:00:00 was successful " was wrong. It was just queued during this time and failed again. |
|
|
Sync's for the Rocky 9.1 BaseOS are working again. Thank you so much!!!! |
|
|
looks for me good too! |
|
|
Good to hear! I appreciate the stack traces also, this helps us as well. We'll leave this ticket open for a little bit longer as everything syncs. |
- Anonymous
