|
View Issue Details |
|
|
ID: |
Category: |
Severity: |
Reproducibility: |
Date Submitted: |
Last Update: |
|
2576 |
[Rocky-Linux-9] kernel |
minor |
always |
2023-03-07 14:59 |
2023-03-07 14:59 |
|
|
Reporter: |
Tadas Sutkaitis |
Platform: |
|
|
|
Assigned To: |
|
OS: |
|
|
|
Priority: |
normal |
OS Version: |
|
|
|
Status: |
new |
Product Version: |
|
|
|
Product Build: |
|
Resolution: |
open |
|
|
Projection: |
none |
|
|
|
|
ETA: |
none |
Fixed in Version: |
|
|
|
|
|
Target Version: |
|
|
|
|
Summary: |
Haproxy pod keeps OOM crashing |
|
Description: |
Running Kubernetes on Rocky Linux 9 with kernel 5.14, haproxy pod keeps OOM crashing while works fine on Rocky Linux 8 or Ubuntu 20.04.22.04
All the pods are running on four bare metal worker nodes inside a Kubernetes clusters. Two worker nodes have Rocky Linux 8 installed, one Ubuntu 22.04 while one worker nodes has Rocky Linux 9 installed with kernel 5.14. Three of the haproxy pods running on Rocky Linux 8 and Ubuntu 22.04 nodes using memory less than 100MB but that one running on Rocky Linux 9 could use all the memory on the node without limits and easily reaches its memory limits like the one above with 2GB memory limits configured.
I also tried to bump its haproxy version to latest stable version 2.7.2 which didn't help and the pod still keeps being OOMKilled.
I also tried using kernel 6.2.x from elrepo.org |
|
Tags: |
|
|
Steps To Reproduce: |
Use kubelet/kubeadm 1.22.x-1.25.x
Use any version of runc and containerd
Use stock kernel 5.14 or
even 6.2.x from elrepo.org |
|
Additional Information: |
log from kernel 5.14(stock):
[ 152.697004] sshd invoked oom-killer: gfp_mask=0x1140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0
[ 152.697396] CPU: 1 PID: 700 Comm: sshd Tainted: G ---------h--- 5.14.0-162.18.1.el9_1.x86_64 #1
[ 152.697774] Hardware name: VMware, Inc. VMware7,1/440BX Desktop Reference Platform, BIOS VMW71.00V.20904234.B64.2212051115 12/05/2022
[ 152.698539] Call Trace:
[ 152.698918] dump_stack_lvl+0x34/0x48
[ 152.699299] dump_header+0x4a/0x201
[ 152.699676] oom_kill_process.cold+0xb/0x10
[ 152.700054] out_of_memory+0xed/0x2d0
[ 152.700432] __alloc_pages_slowpath.constprop.0+0x7cc/0x8a0
[ 152.700814] __alloc_pages+0x1fe/0x230
[ 152.701191] folio_alloc+0x17/0x50
[ 152.701569] __filemap_get_folio+0x1b6/0x330
[ 152.701947] ? do_sync_mmap_readahead+0x14b/0x270
[ 152.702327] filemap_fault+0x454/0x7a0
[ 152.702706] ? next_uptodate_page+0x160/0x1f0
[ 152.703085] ? filemap_map_pages+0x307/0x4a0
[ 152.703465] __xfs_filemap_fault+0x66/0x280 [xfs]
[ 152.703909] __do_fault+0x33/0x110
[ 152.704289] do_read_fault+0xea/0x190
[ 152.704667] do_fault+0x8c/0x2c0
[ 152.705044] __handle_mm_fault+0x3cb/0x750
[ 152.705424] handle_mm_fault+0xc5/0x2a0
[ 152.705799] do_user_addr_fault+0x1bb/0x690
[ 152.706178] ? do_syscall_64+0x69/0x90
[ 152.706557] exc_page_fault+0x62/0x150
[ 152.706959] asm_exc_page_fault+0x22/0x30
[ 152.707348] RIP: 0033:0x5633f1eadae0
[ 152.707732] Code: Unable to access opcode bytes at RIP 0x5633f1eadab6.
[ 152.708132] RSP: 002b:00007ffc2fae4de8 EFLAGS: 00010206
[ 152.708530] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000008
[ 152.708937] RDX: 0000000000000004 RSI: 00005633f2c1a120 RDI: 00005633f2bf1b60
[ 152.709348] RBP: 00005633f2c1a120 R08: 00005633f2c1a120 R09: 0000000000000000
[ 152.709756] R10: 00007ffc2fbb0080 R11: 0000000000000246 R12: 0000000000000002
[ 152.710164] R13: 00005633f2bf3230 R14: 00005633f2c1a120 R15: 00005633f2bf1b60
[ 152.710603] Mem-Info:
[ 152.711029] active_anon:837 inactive_anon:1940583 isolated_anon:0
active_file:19 inactive_file:29 isolated_file:0
unevictable:0 dirty:0 writeback:0
slab_reclaimable:6301 slab_unreclaimable:15610
mapped:682 shmem:2491 pagetables:4647 bounce:0
kernel_misc_reclaimable:0
free:25244 free_pcp:380 free_cma:0
[ 152.713899] Node 0 active_anon:3348kB inactive_anon:7762332kB active_file:76kB inactive_file:116kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:2728kB dirty:0kB writeback:0kB shmem:9964kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 7049216kB writeback_tmp:0kB kernel_stack:6832kB pagetables:18588kB all_unreclaimable? no
[ 152.715107] Node 0 DMA free:14336kB min:128kB low:160kB high:192kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[ 152.716328] lowmem_reserve[]: 0 2959 7854 7854 7854
[ 152.716718] Node 0 DMA32 free:44784kB min:25412kB low:31764kB high:38116kB reserved_highatomic:0KB active_anon:0kB inactive_anon:3016548kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:3128720kB managed:3063120kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[ 152.717967] lowmem_reserve[]: 0 0 4895 4895 4895
[ 152.718386] Node 0 Normal free:41856kB min:42040kB low:52548kB high:63056kB reserved_highatomic:0KB active_anon:3348kB inactive_anon:4745784kB active_file:428kB inactive_file:220kB unevictable:0kB writepending:0kB present:5242880kB managed:5020764kB mlocked:0kB bounce:0kB free_pcp:1520kB local_pcp:556kB free_cma:0kB
[ 152.719752] lowmem_reserve[]: 0 0 0 0 0
[ 152.720225] Node 0 DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB (M) 3*4096kB (M) = 14336kB
[ 152.720709] Node 0 DMA32: 6*4kB (UM) 4*8kB (UM) 11*16kB (UM) 6*32kB (UM) 5*64kB (UM) 2*128kB (UM) 1*256kB (U) 0*512kB 1*1024kB (M) 1*2048kB (U) 10*4096kB (UM) = 45288kB
[ 152.721706] Node 0 Normal: 455*4kB (UME) 260*8kB (UME) 140*16kB (UME) 61*32kB (UE) 56*64kB (UME) 29*128kB (UME) 23*256kB (UME) 25*512kB (UME) 8*1024kB (UME) 0*2048kB 0*4096kB = 42268kB
[ 152.722762] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[ 152.723309] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[ 152.723834] 2549 total pagecache pages
[ 152.724373] 0 pages in swap cache
[ 152.724895] Swap cache stats: add 0, delete 0, find 0/0
[ 152.725440] Free swap = 0kB
[ 152.725983] Total swap = 0kB
[ 152.726506] 2096898 pages RAM
[ 152.727049] 0 pages HighMem/MovableOnly
[ 152.727572] 72087 pages reserved
[ 152.728111] 0 pages cma reserved
[ 152.728631] 0 pages hwpoisoned
[ 152.729168] Tasks state (memory values in pages):
[ 152.729687] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[ 152.730243] [ 542] 0 542 6490 942 86016 0 -250 systemd-journal
[ 152.730778] [ 558] 0 558 8489 1001 86016 0 -1000 systemd-udevd
[ 152.731330] [ 586] 0 586 4535 192 57344 0 -1000 auditd
[ 152.731861] [ 610] 0 610 19813 70 53248 0 0 irqbalance
[ 152.732413] [ 612] 0 612 41041 1291 90112 0 0 rsyslogd
[ 152.732960] [ 615] 0 615 4572 281 73728 0 0 systemd-logind
[ 152.733486] [ 617] 192 617 1730 118 53248 0 0 systemd-network
[ 152.734026] [ 621] 996 621 21106 166 65536 0 0 chronyd
[ 152.734547] [ 622] 81 622 2705 164 61440 0 -900 dbus-broker-lau
[ 152.735086] [ 624] 81 624 1248 104 45056 0 -900 dbus-broker
[ 152.735606] [ 627] 0 627 4038 395 77824 0 -1000 sshd
[ 152.736172] [ 630] 0 630 1519 215 45056 0 0 crond
[ 152.736694] [ 631] 0 631 761 21 45056 0 0 agetty
[ 152.737241] [ 632] 0 632 485312 10947 421888 0 -999 containerd
[ 152.737749] [ 689] 0 689 4810 561 77824 0 0 sshd
[ 152.738263] [ 692] 0 692 4929 413 81920 0 100 systemd
[ 152.738743] [ 693] 0 693 5543 731 81920 0 100 (sd-pam)
[ 152.739229] [ 700] 0 700 4843 611 77824 0 0 sshd
[ 152.739683] [ 701] 0 701 3495 2413 65536 0 0 bash
[ 152.740141] [ 736] 0 736 485268 9143 499712 0 -999 kubelet
[ 152.740579] [ 829] 0 829 178113 1063 106496 0 -998 containerd-shim
[ 152.741027] [ 855] 0 855 178113 1025 110592 0 -998 containerd-shim
[ 152.741441] [ 893] 0 893 178113 1017 106496 0 -998 containerd-shim
[ 152.741840] [ 899] 0 899 178113 1088 106496 0 -998 containerd-shim
[ 152.742252] [ 928] 0 928 178113 1144 102400 0 -998 containerd-shim
[ 152.742633] [ 932] 65535 932 243 1 28672 0 -998 pause
[ 152.743017] [ 959] 65535 959 243 1 28672 0 -998 pause
[ 152.743370] [ 999] 65535 999 243 1 28672 0 -998 pause
[ 152.743713] [ 1002] 65535 1002 243 1 28672 0 -998 pause
[ 152.744060] [ 1022] 65535 1022 243 1 28672 0 -998 pause
[ 152.744376] [ 1115] 0 1115 241739 45023 671744 0 -997 kube-apiserver
[ 152.744683] [ 1196] 0 1196 2803664 7283 192512 0 -997 etcd
[ 152.744995] [ 1203] 0 1203 188393 3900 204800 0 -997 kube-scheduler
[ 152.745280] [ 1231] 0 1231 205461 9729 364544 0 -997 kube-controller
[ 152.745556] [ 1239] 0 1239 6296 227 81920 0 1000 keepalived
[ 152.745841] [ 1290] 0 1290 6316 242 77824 0 1000 keepalived
[ 152.746131] [ 1548] 0 1548 178113 1198 110592 0 -998 containerd-shim
[ 152.746406] [ 1569] 65535 1569 243 1 28672 0 -998 pause
[ 152.746679] [ 1658] 99 1658 41965078 1835879 14815232 0 1000 haproxy
[ 152.746971] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/containerd.service/kubepods-besteffort-pod4bff3fd65add1f5f664ebc8e39174145.slice:cri-containerd:d0e7d35e48d1d2c1c4c7398b61375e7fec8dffabb84c972d3be2e08dac600439,task=haproxy,pid=1658,uid=99
[ 152.747889] Out of memory: Killed process 1658 (haproxy) total-vm:167860312kB, anon-rss:7343516kB, file-rss:0kB, shmem-rss:0kB, UID:99 pgtables:14468kB oom_score_adj:1000
#####################
log from kernel 6.2:
[ 821.389534] systemd-rc-local-generator[4503]: /etc/rc.d/rc.local is not marked executable, skipping.
[ 821.608848] systemd[1]: systemd 250-12.el9_1.3 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS -FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT -QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[ 821.610556] systemd[1]: Detected virtualization vmware.
[ 821.611494] systemd[1]: Detected architecture x86-64.
[ 821.659963] systemd-rc-local-generator[4523]: /etc/rc.d/rc.local is not marked executable, skipping.
[ 833.885781] systemd-rc-local-generator[5227]: /etc/rc.d/rc.local is not marked executable, skipping.
[ 847.648883] kworker/1:0 invoked oom-killer: gfp_mask=0xdc0(GFP_KERNEL|__GFP_ZERO), order=0, oom_score_adj=0
[ 847.649165] CPU: 1 PID: 1596 Comm: kworker/1:0 Tainted: G E 6.2.2-1.el9.elrepo.x86_64 #1
[ 847.649439] Hardware name: VMware, Inc. VMware7,1/440BX Desktop Reference Platform, BIOS VMW71.00V.20904234.B64.2212051115 12/05/2022
[ 847.649700] Workqueue: events drm_fb_helper_damage_work [drm_kms_helper]
[ 847.649978] Call Trace:
[ 847.650237] <TASK>
[ 847.650493] dump_stack_lvl+0x45/0x5e
[ 847.650758] dump_header+0x4a/0x213
[ 847.651034] oom_kill_process.cold+0xb/0x10
[ 847.651349] out_of_memory+0xed/0x2e0
[ 847.651618] __alloc_pages_slowpath.constprop.0+0x78b/0xb00
[ 847.651880] __alloc_pages+0x21d/0x250
[ 847.652135] vmw_validation_mem_alloc+0x78/0x120 [vmwgfx]
[ 847.652408] vmw_validation_add_resource+0xd6/0x290 [vmwgfx]
[ 847.652677] ? drm_plane_get_damage_clips+0x23/0x60 [drm]
[ 847.652968] vmw_du_helper_plane_update+0x102/0x350 [vmwgfx]
[ 847.653239] vmw_stdu_plane_update_surface+0xc4/0xf0 [vmwgfx]
[ 847.653510] ? __pfx_vmw_stdu_surface_fifo_size_same_display+0x10/0x10 [vmwgfx]
[ 847.653783] ? __pfx_vmw_stdu_surface_update_proxy+0x10/0x10 [vmwgfx]
[ 847.654055] ? __pfx_vmw_stdu_surface_populate_update+0x10/0x10 [vmwgfx]
[ 847.654329] vmw_stdu_primary_plane_atomic_update+0xbb/0x1c0 [vmwgfx]
[ 847.654606] drm_atomic_helper_commit_planes+0xb4/0x220 [drm_kms_helper]
[ 847.654901] drm_atomic_helper_commit_tail+0x45/0x80 [drm_kms_helper]
[ 847.655183] commit_tail+0xd2/0x130 [drm_kms_helper]
[ 847.655462] drm_atomic_helper_commit+0x11b/0x140 [drm_kms_helper]
[ 847.655742] drm_atomic_commit+0x93/0xc0 [drm]
[ 847.656039] ? __pfx___drm_printfn_info+0x10/0x10 [drm]
[ 847.656336] drm_atomic_helper_dirtyfb+0x192/0x270 [drm_kms_helper]
[ 847.656627] drm_fbdev_fb_dirty+0x146/0x1e0 [drm_kms_helper]
[ 847.656918] drm_fb_helper_damage_work+0x87/0x160 [drm_kms_helper]
[ 847.657211] process_one_work+0x1e2/0x3b0
[ 847.657494] ? __pfx_worker_thread+0x10/0x10
[ 847.657775] worker_thread+0x50/0x3a0
[ 847.658054] ? __pfx_worker_thread+0x10/0x10
[ 847.658332] kthread+0xe5/0x110
[ 847.658610] ? __pfx_kthread+0x10/0x10
[ 847.658886] ret_from_fork+0x29/0x50
[ 847.659162] </TASK>
[ 847.659464] Mem-Info:
[ 847.659724] active_anon:978 inactive_anon:1946881 isolated_anon:0
active_file:0 inactive_file:22 isolated_file:0
unevictable:0 dirty:0 writeback:0
slab_reclaimable:6083 slab_unreclaimable:13964
mapped:1894 shmem:4514 pagetables:4753
sec_pagetables:0 bounce:0
kernel_misc_reclaimable:0
free:25293 free_pcp:0 free_cma:0
[ 847.661565] Node 0 active_anon:3912kB inactive_anon:7787524kB active_file:0kB inactive_file:88kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:7576kB dirty:0kB writeback:0kB shmem:18056kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 6526976kB writeback_tmp:0kB kernel_stack:6704kB pagetables:19012kB sec_pagetables:0kB all_unreclaimable? yes
[ 847.662252] Node 0 DMA free:14336kB boost:0kB min:128kB low:160kB high:192kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[ 847.662756] lowmem_reserve[]: 0 2959 7858 7858 7858
[ 847.662996] Node 0 DMA32 free:44964kB boost:0kB min:25400kB low:31748kB high:38096kB reserved_highatomic:0KB active_anon:0kB inactive_anon:3010376kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:3128596kB managed:3063000kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[ 847.663522] lowmem_reserve[]: 0 0 4899 4899 4899
[ 847.663784] Node 0 Normal free:41872kB boost:0kB min:42052kB low:52564kB high:63076kB reserved_highatomic:0KB active_anon:3912kB inactive_anon:4777148kB active_file:16kB inactive_file:216kB unevictable:0kB writepending:0kB present:5242880kB managed:5024212kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[ 847.664639] lowmem_reserve[]: 0 0 0 0 0
[ 847.664925] Node 0 DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB (M) 3*4096kB (M) = 14336kB
[ 847.665228] Node 0 DMA32: 5*4kB (UM) 7*8kB (UM) 8*16kB (UM) 11*32kB (UM) 6*64kB (UM) 4*128kB (UM) 2*256kB (M) 5*512kB (UM) 4*1024kB (M) 0*2048kB 9*4096kB (M) = 45484kB
[ 847.665871] Node 0 Normal: 1117*4kB (UME) 462*8kB (UME) 216*16kB (UME) 57*32kB (UME) 68*64kB (UME) 25*128kB (UE) 8*256kB (UME) 6*512kB (UME) 16*1024kB (UME) 0*2048kB 0*4096kB = 42500kB
[ 847.666547] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[ 847.666878] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[ 847.667203] 4526 total pagecache pages
[ 847.667546] 0 pages in swap cache
[ 847.667869] Free swap = 0kB
[ 847.668188] Total swap = 0kB
[ 847.668522] 2096867 pages RAM
[ 847.668837] 0 pages HighMem/MovableOnly
[ 847.669150] 71224 pages reserved
[ 847.669479] 0 pages cma reserved
[ 847.669789] 0 pages hwpoisoned
[ 847.670095] Tasks state (memory values in pages):
[ 847.670422] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[ 847.670749] [ 536] 0 536 8650 2368 102400 0 -250 systemd-journal
[ 847.671068] [ 551] 0 551 8153 480 86016 0 -1000 systemd-udevd
[ 847.671410] [ 578] 0 578 4535 706 57344 0 -1000 auditd
[ 847.671729] [ 603] 0 603 19813 580 61440 0 0 irqbalance
[ 847.672046] [ 605] 0 605 59473 2208 98304 0 0 rsyslogd
[ 847.672364] [ 606] 0 606 4572 288 69632 0 0 systemd-logind
[ 847.672705] [ 611] 192 611 1730 96 49152 0 0 systemd-network
[ 847.673028] [ 613] 996 613 21106 131 65536 0 0 chronyd
[ 847.673367] [ 616] 81 616 2705 192 61440 0 -900 dbus-broker-lau
[ 847.673692] [ 617] 81 617 1248 128 40960 0 -900 dbus-broker
[ 847.674016] [ 620] 0 620 4038 384 65536 0 -1000 sshd
[ 847.674338] [ 623] 0 623 1520 192 49152 0 0 crond
[ 847.674679] [ 624] 0 624 761 0 40960 0 0 agetty
[ 847.675002] [ 625] 0 625 483312 11430 413696 0 -999 containerd
[ 847.675325] [ 675] 0 675 4810 544 73728 0 0 sshd
[ 847.675669] [ 678] 0 678 4918 384 77824 0 100 systemd
[ 847.675993] [ 679] 0 679 5541 730 69632 0 100 (sd-pam)
[ 847.676337] [ 686] 0 686 4810 537 73728 0 0 sshd
[ 847.676663] [ 687] 0 687 1512 480 49152 0 0 bash
[ 847.676980] [ 2026] 0 2026 4810 576 73728 0 0 sshd
[ 847.677289] [ 2028] 0 2028 4842 604 69632 0 0 sshd
[ 847.677608] [ 4462] 0 4462 4374 224 73728 0 0 systemd-hostnam
[ 847.677902] [ 5142] 0 5142 1149 64 45056 0 0 sh
[ 847.678186] [ 5154] 0 5154 6179 2967 81920 0 0 python3
[ 847.678483] [ 5155] 0 5155 187580 3342 204800 0 0 kubeadm
[ 847.678758] [ 5237] 0 5237 427856 7811 466944 0 -999 kubelet
[ 847.679023] [ 5304] 0 5304 180044 1246 106496 0 -998 containerd-shim
[ 847.679282] [ 5328] 0 5328 180044 1226 106496 0 -998 containerd-shim
[ 847.679552] [ 5339] 0 5339 180044 1237 110592 0 -998 containerd-shim
[ 847.679793] [ 5345] 0 5345 180044 1254 114688 0 -998 containerd-shim
[ 847.680024] [ 5359] 0 5359 180044 1255 102400 0 -998 containerd-shim
[ 847.680253] [ 5369] 65535 5369 243 0 28672 0 -998 pause
[ 847.680497] [ 5361] 0 5361 180044 1755 110592 0 -998 containerd-shim
[ 847.680718] [ 5416] 65535 5416 243 0 28672 0 -998 pause
[ 847.680927] [ 5466] 65535 5466 243 0 28672 0 -998 pause
[ 847.681125] [ 5467] 65535 5467 243 0 28672 0 -998 pause
[ 847.681317] [ 5492] 65535 5492 243 32 28672 0 -998 pause
[ 847.681518] [ 5500] 65535 5500 243 0 28672 0 -998 pause
[ 847.681701] [ 5564] 0 5564 188393 3323 208896 0 -997 kube-scheduler
[ 847.681882] [ 5615] 0 5615 241579 40190 630784 0 -997 kube-apiserver
[ 847.682063] [ 5618] 0 5618 6296 192 81920 0 1000 keepalived
[ 847.682246] [ 5648] 0 5648 205333 5527 339968 0 -997 kube-controller
[ 847.682454] [ 5655] 0 5655 2803664 6184 180224 0 -997 etcd
[ 847.682643] [ 5696] 0 5696 6316 219 81920 0 1000 keepalived
[ 847.682832] [ 5739] 99 5739 41965116 1845247 14876672 0 1000 haproxy
[ 847.683021] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/k8s.io/e53c4abf72ff20acdbd7a50818b95ef02dfc5401f0c4515368104c487be65d7d,task=haproxy,pid=5739,uid=99 |
|
Attached Files: |
|
|
|
Notes |
(0000487)
Louis Abel
2022-08-30 21:20
|
Hello. Please try the following images:
http://dl.rockylinux.org/stg/rocky/9/images/x86_64/Rocky-9-GenericCloud-9.0-20220830.0.x86_64.qcow2
http://dl.rockylinux.org/stg/rocky/9/images/x86_64/Rocky-9-GenericCloud.latest.x86_64.qcow2 |
|
(0000491)
Jean-François Landry
2022-08-31 15:57
|
Looks fine aside from a small locale / glibc langpack issue.
According to localectl: System Locale: LANG=en_US.UTF-8
Installed glibc langpacks: glibc-minimal-langpack-2.34-28.el9_0.x86_64
Which results in warning when running sudo dnf update and likely other commands: "Failed to set locale, defaulting to C.UTF-8"
Can be fixed by installing "glibc-langpack-en" or using "LANG=C.UTF-8" by default.
Otherwise, the image boots in BIOS mode on openstack, cloud-init/growpart work, seems totally functional. |
|
(0000492)
Robert Cohen
2022-09-01 02:53
|
trivial issue.
Since this image is for use in virtual environments, it would be nice to have the virt-what package installed by default since some software uses it to detect
that this is a virtual environment. |
|
(0000533)
Fritz Elfert
2022-09-14 11:59
|
In addition to the missing glibc-langpack-en mentioned earlier, I just noticed another odd
behavior with the new image Rocky-9-GenericCloud-9.0-20220830.0.x86_64.qcow2:
With the old image, output of cloud-init was logged to the console (/dev/ttyS0) as well as to /var/log/cloud-init-output.log
With the new image, logging of cloud-init on the console is gone.
This affects automated provsioning of custom snapshots here, where we monitor a VMs console for specific patterns
in order to detect errors in an early stage and then prevent out automated provisioning process from createing a
possibly broken snapshot.
I just tested current Alma9 in regards to that and it still produces cloud-init output on the console. Since both provide
the same cloud-init package version, I assume this is not an issue of cloud-init itself but some side effect of a generic
configuration change. Will test the same with rhel-baseos-9.0-x86_64-kvm.qcow2 from RedHat later.
PS:
I would look into your image-creation process myself, but I can't find any public (git?) repo. The only one that
I found is https://github.com/rocky-linux/sig-cloud-instance-images but that one appears to be for docker.
So: Where the heck are the scripts that you use for generating the official images? |
|
(0000534)
Neil Hanlon
2022-09-14 13:20
|
Working on new images this week which will address these concerns. I'll have to look into the cloud-init one a bit more, but that shouldn't be too hard. I suspect something going on with the console parameter.
@Fritz regarding the scripts for image generation. we're moving things into our toolkit in a subproject. Specifically ,the image building script is here: https://git.resf.org/sig_core/toolkit/src/branch/devel/iso/empanadas/empanadas/scripts/build_image.py
I need to work on some docs for usage, but largely that's how the images get generated in our semi-hand-fed pipeline. |
|
(0000536)
Louis Abel
2022-09-15 03:32
|
Hello.
All kickstarts are here: https://git.resf.org/sig_core/kickstarts or the mirror here: https://github.com/rocky-linux/kickstarts
The langpack* packages have always been excluded in our images since we started making them. This was a carry over from how CentOS does it (including in stream 9). We usually remove and add packages that we may feel that do or don't fit as well.
For context with the locale piece: As far as I'm aware, the locale piece is actually a bug of cloud init that should be fixed in 22.2 (or a patched 22.1 that's currently in CentOS Stream 9). I did a hotfix for the next image we build to force the locale to be C.UTF-8 in the meantime. See this commit: https://git.resf.org/sig_core/kickstarts/commit/ebf7a38c6b3f81512daaf0f4938148c1a576750b
As for the console portion, it's a strange case. We have tried to adjust the parameters in the kickstart bootloader line and we have also tried to use sed and grubby within %post to try to resolve the problem. What ends up happening with the changes we try is we no longer get any console on ttyS0, not even a login prompt. We have also tried to inherit a fedora-like configuration specifying `console=tty1 console=ttyS0,115200n8` at the end of the bootloader configuration, but still no dice. Even more interesting, you can run a few grubby commands after logging in to "fix" the issue, but it defeats the purpose of not being able to see the cloud init messages on first boot.
As an aside, if you would like to try out how we build our images, you could try this (it may require you to make some directories or install packages on your host system; we also use Fedora in this example):
git clone https://git.resf.org/sig_core/toolkit.git -b devel
cd toolkit/iso/empanadas
podman build --build-arg BRANCH=r9 -t empanadas:r9 -f Containerfile.imagefactory
podman run -it --rm -e LIBVIRT_DEFAULT_URI -v /var/run/libvirt/:/var/run/libvirt/ -v /var/lib/imagefactory/:/var/lib/imagefactory/:rw --network host --security-opt label=disable --privileged --device fuse empanadas:r9 build-image --version 9 --type GenericCloud --debug
This will leave an image in /var/lib/imagefactory/storage with a .body name. It'll be in raw format. You can then use this against virt-install or otherwise.
I'll keep trying to look into where the issue is in the meanwhile. We are open to any suggestions or help as well to try to resolve the console issue. |
|
(0000537)
Louis Abel
2022-09-15 06:38
|
After a bit of work we believe we have the issues resolved. We'll have images built through the week/weekend and pushed to our mirrors and we'll notify when this happens.
Resolved in the following commits:
Remove kernel parameters from icicle configuration: https://git.resf.org/sig_core/toolkit/commit/eeae7ff5ab73f27229bbc578aeeb3750619101b9
Fixes grub2-pc package naming and removes sed in %post: https://git.resf.org/sig_core/kickstarts/commit/0a16a1b5cf1b3fbfe6bc8b95fcf19b645fef0389
Works around cloud-init bug that should be backported in 9.1's cloud-init or fixed in 22.2: https://git.resf.org/sig_core/kickstarts/commit/ebf7a38c6b3f81512daaf0f4938148c1a576750b
Adds virt-what to package list: https://git.resf.org/sig_core/kickstarts/commit/008bf056b923231c0f9100fabef54d2d407c4304 |
|
|
Notes |
(0002377)
Ashutosh Kumar
2023-02-09 20:19
|
Just wanted to mention that Fedora version was 36 and the Ubuntu version was 22.4.1 and Rocky Linux was updated completely as of 7 PM of10th February 2023. |
|
(0002443)
Ashutosh Kumar
2023-02-10 18:28
|
Hello, back here again. I tried kernel version 6.1.8 (kernel-ml) from elrepo for el9 and the borders do not show up! So there is probably some miscofiguration in the kernel |
|
(0002444)
Lukas Magauer
2023-02-10 20:38
|
Hi Ashutosh,
I tried to reproduce this in a freshly installed Rocky Linux 9 machine (ESXi hosted VM),
but I couldn't how did you install Google Chrome and how does your setup look like? |
|
(0002445)
Lukas Magauer
2023-02-10 20:40
|
Or is this only happening if you access the machine via RDP (as far as I can see it should be accessed by RDP) |
|
(0002446)
Ashutosh Kumar
2023-02-10 21:16
|
Hello Lukas, I installed chrome both via flathub and the official google-chrome-stable repo. Both had the same issue. Also, it used to happen when I had Rocky installed on my laptop. This is a Hyper-V VM.
My setup in the image is just a pure Rocky install with workstation install and Rocky icons disabled. Also i had changed adwaita to dark using gnome tweaks.
My laptop is an IdeaPad 5 Pro 14ACN6 with amd Ryzen 7 5800U with integrated graphics.
This doesn't happen with firefox or any other app, but a kernel update from elrepo fixes it. |
|
(0002447)
Lukas Magauer
2023-02-10 21:27
|
Okay very interesting (thank you for the hint I completely forgot about the possibility the RDP heady being a Hyper-V session), so I made a screenshot of my test machine now, the left window is from the 1st party rpm and the right one from the flatpak from flathub. |
|
(0002448)
Lukas Magauer
2023-02-10 21:35
|
Or making it very apparent with Adwaita dark, the flatpak not handling the dark mode ^^
Up to now everything looks good here,
what do you mean by disabling Rocky icons?
(if nothing helps I will pull up a another machine on Hyper-V)
I'm on kernel 5.14.0-162.12.1.el9_1.0.2 right now btw. |
|
(0002449)
Ashutosh Kumar
2023-02-11 05:47
|
Hello, by Rocky icons i meant the logo extension Rocky has on by default to show the logo on wallpaper. Also, I made a video showing the problem (I hope yt links are allowed): https://youtu.be/VsFtItlKUVc
I used the flathub version of chrome for the video.
I tried with the elrepo and the original kernel an you can see that all Rocky provided kernels are having the same border. This issue also happens inside virtualbox and on real machines when Rocky is installed on them. |
|
(0002450)
Ashutosh Kumar
2023-02-11 20:08
|
Just to add, this is also occuring in AlmaLinux. So this probably an RHEL issue, since both of these distros are bug-for-bug compatible. |
|
(0002451)
Lukas Magauer
2023-02-11 20:11
|
Could you spin up a CentOS Stream machine and look if it's also happening there?
If no then it should be fixed by the next minor release |
|
(0002452)
Akemi Yagi
2023-02-11 20:21
|
If it is an upstream issue (RHEL or CentOS Stream), it should be reported to bugzilla.redhat.com. Use this link:
https://bugzilla.redhat.com/enter_bug.cgi?product=Red%20Hat%20Enterprise%20Linux%209&version=CentOS%20Stream |
|
(0002453)
Ashutosh Kumar
2023-02-11 21:29
|
Ok, I will try using CentOS Stream in 7-8 hours and I will also report it in rhel bugzilla. Thanks! |
|
|
Notes |
(0002311)
Thomas Menard
2023-01-31 08:05
|
I just wanted to add my voice to this issue.
We are too using pulp for rocky linux repo mirroring and encounter the same issue.
This is really impacting us (no new server install for now) as we don't use versionning feature on pulp.
Maybe minor is not the right severity....
Thanks |
|
(0002312)
Louis Abel
2023-01-31 08:15
|
I would argue that the error being reported from pulp is incorrect. It has nothing to do with "out of order" metadata and more so the metadata itself being inconsistent.
Between two pieces of the metadata, there are shared bits such as checksums. Unfortunately, you did not provide the error output you were seeing. But it is likely you were seeing two sha sums that were stated as "out of order" when instead the problem is that the sums are not matching between the primaries and file lists. This should be fixed and a sync should work normally.
If you are syncing from a mirror, please give them time to sync. If you are syncing directly from dl.rl.o, it should work right away, assuming fastly cache has been purged in your respective pops. |
|
(0002313)
Holger Hees
2023-01-31 08:42
|
Message coming from pulp is "Msg: Parsing interrupted: Out of order metadata"
I sync every hour from https://ftp.gwdg.de/pub/linux/rocky/9.1/BaseOS/x86_64/os/
The error happens from "Jan 27 07:00:00" until "Jan 31 08:00:00"
Means the last sync at 09:00:00 was successful |
|
(0002314)
Holger Hees
2023-01-31 08:47
|
fullstack trace from pulp is
---
pulp [579b6c3ff3a445f39715b8a9b722e74c]: pulpcore.tasking.pulpcore_worker:INFO: File "/usr/local/lib/python3.8/site-packages/pulpcore/tasking/pulpcore_worker.py", line 452, in _perform_task
result = func(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 565, in synchronize
repo_version = dv.create() or repo.latest_version()
File "/usr/local/lib/python3.8/site-packages/pulpcore/plugin/stages/declarative_version.py", line 161, in create
loop.run_until_complete(pipeline)
File "/usr/lib64/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/usr/local/lib/python3.8/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
await asyncio.gather(*futures)
File "/usr/local/lib/python3.8/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
await self.run()
File "/usr/local/lib/python3.8/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 856, in run
await self.parse_repository_metadata(repomd, repomd_files)
File "/usr/local/lib/python3.8/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 908, in parse_repository_metadata
await self.parse_packages(
File "/usr/local/lib/python3.8/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 1281, in parse_packages
for pkg in parser.as_iterator(): |
|
(0002315)
Holger Hees
2023-01-31 08:48
|
I missed the first line from the stack trace
pulp [579b6c3ff3a445f39715b8a9b722e74c]: pulpcore.tasking.pulpcore_worker:INFO: Task 4f5a132d-1624-42e0-8d5f-1f6163b1c622 failed (Parsing interrupted: Out of order metadata: d2134ac3dc9c0b26098b6e5d50a51f2a47df3bd8b4801369f39a815c0cfdba10 vs cf36b2caa37bf0e45b7c3d456a4897c6ab645367e5286c28bcac77b92d6c9d04. |
|
(0002316)
Thomas Menard
2023-01-31 09:03
|
Here is our pulp output
```
an 30 13:29:53 pulp01.it.pasteur.fr pulpcore-worker[3001341]: pulp [d104d89f04454a76acbc94bbdf02e535]: pulpcore.tasking.pulpcore_worker:INFO: Task b94008d7-f7d3-4ac4-91cd-17169f4ede02 failed (Parsing interrupted: Out of order metadata: 786010ce275e9a122265d52515eb90784e036bdf68f2ed2d78d1de250016d3f0 vs be8f8f34a444f82e2d057d799e1eaf4c69197efeb1a46ecd2d4cc1a7ea37949f.)
Jan 30 13:29:53 pulp01.it.pasteur.fr pulpcore-worker[3001341]: pulp [d104d89f04454a76acbc94bbdf02e535]: pulpcore.tasking.pulpcore_worker:INFO: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulpcore/tasking/pulpcore_worker.py", line 452, in _perform_task
Jan 30 13:29:53 pulp01.it.pasteur.fr pulpcore-worker[3001341]: result = func(*args, **kwargs)
Jan 30 13:29:53 pulp01.it.pasteur.fr pulpcore-worker[3001341]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 565, in synchronize
Jan 30 13:29:53 pulp01.it.pasteur.fr pulpcore-worker[3001341]: repo_version = dv.create() or repo.latest_version()
Jan 30 13:29:53 pulp01.it.pasteur.fr pulpcore-worker[3001341]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulpcore/plugin/stages/declarative_version.py", line 161, in create
Jan 30 13:29:53 pulp01.it.pasteur.fr pulpcore-worker[3001341]: loop.run_until_complete(pipeline)
Jan 30 13:29:53 pulp01.it.pasteur.fr pulpcore-worker[3001341]: File "/usr/lib64/python3.9/asyncio/base_events.py", line 642, in run_until_complete
Jan 30 13:29:53 pulp01.it.pasteur.fr pulpcore-worker[3001341]: return future.result()
Jan 30 13:29:53 pulp01.it.pasteur.fr pulpcore-worker[3001341]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
Jan 30 13:29:53 pulp01.it.pasteur.fr pulpcore-worker[3001341]: await asyncio.gather(*futures)
Jan 30 13:29:53 pulp01.it.pasteur.fr pulpcore-worker[3001341]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
Jan 30 13:29:53 pulp01.it.pasteur.fr pulpcore-worker[3001341]: await self.run()
Jan 30 13:29:53 pulp01.it.pasteur.fr pulpcore-worker[3001341]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 856, in run
Jan 30 13:29:53 pulp01.it.pasteur.fr pulpcore-worker[3001341]: await self.parse_repository_metadata(repomd, repomd_files)
Jan 30 13:29:53 pulp01.it.pasteur.fr pulpcore-worker[3001341]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 908, in parse_repository_metadata
Jan 30 13:29:53 pulp01.it.pasteur.fr pulpcore-worker[3001341]: await self.parse_packages(
Jan 30 13:29:53 pulp01.it.pasteur.fr pulpcore-worker[3001341]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 1281, in parse_packages
Jan 30 13:29:53 pulp01.it.pasteur.fr pulpcore-worker[3001341]: for pkg in parser.as_iterator():
``` |
|
(0002318)
Holger Hees
2023-01-31 10:37
|
my sentence "Means the last sync at 09:00:00 was successful " was wrong. It was just queued during this time and failed again. |
|
(0002320)
Matt Norwood
2023-01-31 14:34
|
Sync's for the Rocky 9.1 BaseOS are working again. Thank you so much!!!! |
|
(0002321)
Holger Hees
2023-01-31 19:23
|
looks for me good too! |
|
(0002322)
Louis Abel
2023-01-31 19:27
|
Good to hear! I appreciate the stack traces also, this helps us as well. We'll leave this ticket open for a little bit longer as everything syncs. |
|
|
View Issue Details |
|
|
ID: |
Category: |
Severity: |
Reproducibility: |
Date Submitted: |
Last Update: |
|
279 |
[Cloud] General |
major |
always |
2022-09-12 23:35 |
2023-01-24 06:43 |
|
|
Reporter: |
Adam Augustine |
Platform: |
Amazon AMI |
|
|
Assigned To: |
Neil Hanlon |
OS: |
Rocky Linux x86_64 |
|
|
Priority: |
normal |
OS Version: |
8.6 |
|
|
Status: |
confirmed |
Product Version: |
|
|
|
Product Build: |
|
Resolution: |
open |
|
|
Projection: |
none |
|
|
|
|
ETA: |
none |
Fixed in Version: |
|
|
|
|
|
Target Version: |
|
|
|
|
Summary: |
Rocky Linux AMI instances fail to configure network properly when in IPv6-only subnet in AWS |
|
Description: |
I am having trouble launching official Rocky AMI instances in AWS IPv6-only subnets. I have tried booting an Amazon Linux AMI and it works properly. I have tested 8.6, 9.0, x86_64 and aarch64.
The instances will take many minutes to boot, throw many error messages, and will eventually fail the second "Status Check" with the error message "Instance reachability check failed".
Here is the first system log message that differs substantially from the working Amazon AMI (with one line before and after for context):
[ 10.676733] cloud-init[823]: Cloud-init v. 21.1-15.el8 running 'init-local' at Mon, 12 Sep 2022 20:05:47 +0000. Up 10.34 seconds.
[ 10.681192] cloud-init[823]: 2022-09-12 20:05:48,274 - util.py[WARNING]: Getting data from <class 'cloudinit.sources.DataSourceEc2.DataSourceEc2Local'> failed
[[0;32m OK [0m] Started Initial cloud-init job (pre-networking).
The cloud-init process normally (based on the working Amazon AMI instance and other Rocky instances tested in a dual stack subnet) logs an "IPv4 Route Table" with many 169.254.x.y routes. On the failing Rocky instances no "IPv4 Route Table" is logged (see attached system logs).
In the system logs there are also entries reporting attempts to access the assets and metadata server at 169.254.169.254:
[ 12.865186] cloud-init[899]: 2022-09-12 20:05:50,230 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/latest/api/token' failed [0/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe4404f8128>: Failed to establish a new connection: [Errno 101] Network is unreachable',))]
[ 13.648505] cloud-init[899]: 2022-09-12 20:05:51,234 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/latest/api/token' failed [1/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe4404f8a20>: Failed to establish a new connection: [Errno 101] Network is unreachable',))]
These messages repeat many times before the final error messages:
[ 131.835862] cloud-init[899]: 2022-09-12 20:07:49,433 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/latest/api/token' failed [119/120s]: unexpected error [Attempted to set connect timeout to 0.0, but the timeout cannot be set to a value less than or equal to 0.]
[ 138.843417] cloud-init[899]: 2022-09-12 20:07:56,440 - DataSourceEc2.py[WARNING]: IMDS's HTTP endpoint is probably disabled
The instance then continues to boot and eventually presents a login prompt, however the hostname is simply "localhost" and there is no network connectivity. Eventually the error message "Instance reachability check failed" is reported on the "Status Check" tab of the instance page. |
|
Tags: |
|
|
Steps To Reproduce: |
1) Create a VPC with and IPv6 CIDR block (using either your own or Amazon's IPv6 address space)
2) Create an IPv6 only subnet by creating a new subnet and checking the "IPv6 Only" box
3) Create a Rocky Linux instance and associate it with the IPv6 capable VPC and the IPv6-only subnet.
4) After approximately 10 minutes, the instance will complete the boot process, but will have "1/2 checks passed" in the "Status Check" column, and "Instance reachability check failed" in the "Status Check" tab of the instance details section.
5) The box will not be connected to the network
Let me know if you want more specific details. The main bit is to have an IPv6 only subnet. |
|
Additional Information: |
I compared the /usr/lib/python3.6/site-packages/cloudinit/util.py (82200 bytes) on a Rocky 8.6 instance in a dual stack subnet vs
/usr/lib/python2.7/site-packages/cloudinit/util.py (92995 bytes) on the Amazon AMI instance in the IPv6 only subnet.
The Amazon version is not only larger but has additional copyright and author lines, which indicate to me that they are likely using a later version. Fixing the IPv6-only boot problem may just be as simple as updating to a later version of the cloud-init packages. Though I don't know that for certain (I don't know a lot about creating AMIs), it seemed a reasonable place to start.
I have attached the system logs from various attempts, including a working Amazon AMI instance log, and three logs from non-working Rocky AMI instances.
Let me know if you want anything additional. |
|
Attached Files: |
Working-AWS-AMI.log (50,547 bytes) 2022-09-12 23:35
https://bugs.rockylinux.org/file_download.php?file_id=199&type=bug
non-working-Rocky8.6-x86-64.log (65,535 bytes) 2022-09-12 23:35
https://bugs.rockylinux.org/file_download.php?file_id=200&type=bug
non-working-Rocky9.0-x86-64.log (45,417 bytes) 2022-09-12 23:35
https://bugs.rockylinux.org/file_download.php?file_id=201&type=bug
non-working-Rocky8.6-aarch64.log (26,956 bytes) 2022-09-12 23:35
https://bugs.rockylinux.org/file_download.php?file_id=202&type=bug |
|
|
Notes |
(0001222)
Robert Sjöblom
2022-11-23 09:30
|
Restarting rngd after package SELinux has been upgraded succeeds |
|
(0001255)
Robert Sjöblom
2022-11-24 08:52
|
In some cases, restarting rngd fails with "can't find user daemon"; rngd is prevented from reading the file by selinux. Solution is to reinstall selinux-policy package, then restart rngd. |
|
(0001256)
Louis Abel
2022-11-24 09:39
|
I am unable to replicate this issue. After patching and rebooting an 8.6 system, rngd starts up as expected and there are no selinux errors. Reinstalling the selinux-policy package leads me to believe there's either a configuration issue or there is possibly an edge case you've ran into. Can you provide your entire dnf.rpm.log of the day you ran dnf update?
[root@router scsi]# uname -r
4.18.0-372.26.1.el8_6.x86_64
[root@router scsi]# dnf update -y -q
[root@router scsi]# rpm -q rng-tools
rng-tools-6.15-1.el8.x86_64
[root@router scsi]# rpm -q selinux-policy
selinux-policy-3.14.3-108.el8.noarch
[root@router scsi]# systemctl status rngd
● rngd.service - Hardware RNG Entropy Gatherer Daemon
Loaded: loaded (/usr/lib/systemd/system/rngd.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2022-11-15 11:06:42 MST; 1 weeks 1 days ago
Main PID: 333967 (rngd)
Tasks: 5 (limit: 409620)
Memory: 1.9M
CGroup: /system.slice/rngd.service
└─333967 /usr/sbin/rngd -f --fill-watermark=0 -x pkcs11 -x nist -D daemon:daemon
Nov 15 11:06:42 router systemd[1]: Started Hardware RNG Entropy Gatherer Daemon.
Nov 15 11:06:42 router rngd[333967]: Disabling 7: PKCS11 Entropy generator (pkcs11)
Nov 15 11:06:42 router rngd[333967]: Disabling 5: NIST Network Entropy Beacon (nist)
Nov 15 11:06:42 router rngd[333967]: Initializing available sources
Nov 15 11:06:42 router rngd[333967]: [hwrng ]: Initialized
Nov 15 11:06:42 router rngd[333967]: [rdrand]: Initialization Failed
Nov 15 11:06:42 router rngd[333967]: [jitter]: Initializing AES buffer
Nov 15 11:06:47 router rngd[333967]: [jitter]: Enabling JITTER rng support
Nov 15 11:06:47 router rngd[333967]: [jitter]: Initialized
Nov 15 11:06:47 router rngd[333967]: Process privileges have been dropped to 2:2
[root@router scsi]# init 6
[root@router ~]# systemctl status rngd
● rngd.service - Hardware RNG Entropy Gatherer Daemon
Loaded: loaded (/usr/lib/systemd/system/rngd.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2022-11-24 02:15:32 MST; 1min 15s ago
Main PID: 1537 (rngd)
Tasks: 5 (limit: 409712)
Memory: 6.5M
CGroup: /system.slice/rngd.service
└─1537 /usr/sbin/rngd -f --fill-watermark=0 -x pkcs11 -x nist -D daemon:daemon
Nov 24 02:15:32 router systemd[1]: Started Hardware RNG Entropy Gatherer Daemon.
Nov 24 02:15:32 router rngd[1537]: Disabling 7: PKCS11 Entropy generator (pkcs11)
Nov 24 02:15:32 router rngd[1537]: Disabling 5: NIST Network Entropy Beacon (nist)
Nov 24 02:15:32 router rngd[1537]: Initializing available sources
Nov 24 02:15:32 router rngd[1537]: [hwrng ]: Initialized
Nov 24 02:15:32 router rngd[1537]: [rdrand]: Initialization Failed
Nov 24 02:15:32 router rngd[1537]: [jitter]: Initializing AES buffer
Nov 24 02:15:37 router rngd[1537]: [jitter]: Enabling JITTER rng support
Nov 24 02:15:37 router rngd[1537]: [jitter]: Initialized
Nov 24 02:15:37 router rngd[1537]: Process privileges have been dropped to 2:2
[root@router ~]# audit2why < /var/log/audit/audit.log
[root@router ~]#
[root@router ~]# grep -En 'rng|selinux' /tmp/dnf.rpm.log
5:2022-11-24T01:56:03-0700 SUBDEBUG Upgrade: libselinux-2.9-6.el8.x86_64
44:2022-11-24T01:56:08-0700 SUBDEBUG Upgrade: libselinux-utils-2.9-6.el8.x86_64
95:2022-11-24T01:56:19-0700 SUBDEBUG Upgrade: libselinux-devel-2.9-6.el8.x86_64
144:2022-11-24T01:57:05-0700 SUBDEBUG Upgrade: python3-libselinux-2.9-6.el8.x86_64
196:2022-11-24T01:57:21-0700 SUBDEBUG Upgrade: rpm-plugin-selinux-4.14.3-24.el8_7.x86_64
197:2022-11-24T01:57:21-0700 SUBDEBUG Upgrade: selinux-policy-3.14.3-108.el8.noarch
198:2022-11-24T01:57:39-0700 SUBDEBUG Upgrade: selinux-policy-targeted-3.14.3-108.el8.noarch
247:2022-11-24T01:58:38-0700 SUBDEBUG Upgrade: ipa-selinux-4.9.10-3.module+el8.7.0+1074+aae18f3a.noarch
398:2022-11-24T02:05:30-0700 SUBDEBUG Upgrade: rng-tools-6.15-1.el8.x86_64
522:2022-11-24T02:06:20-0700 SUBDEBUG Upgraded: libselinux-devel-2.9-5.el8.x86_64
540:2022-11-24T02:06:22-0700 SUBDEBUG Upgraded: ipa-selinux-4.9.8-6.module+el8.6.0+797+07647629.noarch
582:2022-11-24T02:06:42-0700 SUBDEBUG Upgraded: rng-tools-6.14-6.git.b2b7934e.el8_6.x86_64
814:2022-11-24T02:08:36-0700 SUBDEBUG Upgraded: rpm-plugin-selinux-4.14.3-24.el8_6.x86_64
815:2022-11-24T02:08:36-0700 SUBDEBUG Upgraded: selinux-policy-targeted-3.14.3-95.el8_6.4.noarch
816:2022-11-24T02:08:37-0700 SUBDEBUG Upgraded: selinux-policy-3.14.3-95.el8_6.4.noarch
842:2022-11-24T02:08:39-0700 SUBDEBUG Upgraded: python3-libselinux-2.9-5.el8.x86_64
848:2022-11-24T02:08:39-0700 SUBDEBUG Upgraded: libselinux-utils-2.9-5.el8.x86_64
910:2022-11-24T02:08:56-0700 SUBDEBUG Upgraded: libselinux-2.9-5.el8.x86_64 |
|
(0001288)
Robert Sjöblom
2022-11-25 14:55
|
We've only seen this issue on a few of the servers in the fleet, namely the postgres servers. We're currently running ~900 postgres servers, out of ~1600 machines total. Around 10 of them have been affected by this issue, so it seems likely that it's an edge condition of some kind. We use ansible to configure all our servers, and so they should be exactly similar in configuration.
The servers we have seen this issue on were initially installed with CentOS 8, and later converted to Rocky using Rocky's conversion script. It's possible that it's related, but it's also a fact that the large majority of servers are in this state. We have much fewer new servers with a clean Rocky 8 install.
When looking into the dnf rpm log during the upgrade window, we found an SELinux policy error due to a memory allocation failure.
2022-11-24T05:43:00+0100 SUBDEBUG Upgrade: selinux-policy-targeted-3.14.3-108.el8.noarch
2022-11-24T05:43:38+0100 INFO SELinux: Could not load policy file /etc/selinux/targeted/policy/policy.31: Cannot allocate memory
2022-11-24T05:43:38+0100 SUBDEBUG Upgrade: sssd-dbus-2.7.3-4.el8_7.1.x86_64
The system journal at the same time shows the following:
Nov 24 05:43:37 hostname kernel: load_policy: page allocation failure: order:4, mode:0x60c0c0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null),cpuset=/,mems_allowed=0
Nov 24 05:43:37 hostname kernel: CPU: 1 PID: 2351056 Comm: load_policy Kdump: loaded Not tainted 4.18.0-372.9.1.el8.x86_64 #1
Nov 24 05:43:37 hostname kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
Nov 24 05:43:37 hostname kernel: Call Trace:
Nov 24 05:43:37 hostname kernel: dump_stack+0x41/0x60
Nov 24 05:43:37 hostname kernel: warn_alloc.cold.119+0x7b/0x111
Nov 24 05:43:37 hostname kernel: ? _cond_resched+0x15/0x30
Nov 24 05:43:37 hostname kernel: ? __alloc_pages_direct_compact+0x157/0x160
Nov 24 05:43:37 hostname kernel: __alloc_pages_slowpath+0xc7e/0xcc0
Nov 24 05:43:37 hostname kernel: ? type_read+0x160/0x160
Nov 24 05:43:37 hostname kernel: __alloc_pages_nodemask+0x2db/0x310
Nov 24 05:43:37 hostname kernel: kmalloc_order+0x28/0x90
Nov 24 05:43:37 hostname kernel: kmalloc_order_trace+0x1d/0xa0
Nov 24 05:43:37 hostname kernel: ? type_read+0x160/0x160
Nov 24 05:43:37 hostname kernel: __kmalloc+0x1ff/0x250
Nov 24 05:43:37 hostname kernel: ? type_read+0x160/0x160
Nov 24 05:43:37 hostname kernel: hashtab_init+0x5d/0x80
Nov 24 05:43:37 hostname kernel: policydb_read+0x2e3/0x1230
Nov 24 05:43:37 hostname kernel: security_load_policy+0xa8/0x5e0
Nov 24 05:43:37 hostname kernel: ? copy_user_generic_unrolled+0x32/0xc0
Nov 24 05:43:37 hostname kernel: sel_write_load+0xde/0x1a0
Nov 24 05:43:37 hostname kernel: vfs_write+0xa5/0x1a0
Nov 24 05:43:37 hostname kernel: ksys_write+0x4f/0xb0
Nov 24 05:43:37 hostname kernel: do_syscall_64+0x5b/0x1a0
Nov 24 05:43:37 hostname kernel: entry_SYSCALL_64_after_hwframe+0x65/0xca
Nov 24 05:43:37 hostname kernel: RIP: 0033:0x7fa9bd1f2bc8
Nov 24 05:43:37 hostname kernel: Code: 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 55 4b 2a 00 8b 00 85 c0 75 17 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 58 c3 0f 1f 80 00 00 00 00 41 54 49 89 d4 55
Nov 24 05:43:37 hostname kernel: RSP: 002b:00007ffe2428f228 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
Nov 24 05:43:37 hostname kernel: RAX: ffffffffffffffda RBX: 00007ffe2428f230 RCX: 00007fa9bd1f2bc8
Nov 24 05:43:37 hostname kernel: RDX: 0000000000833419 RSI: 00007fa9af240000 RDI: 0000000000000004
Nov 24 05:43:37 hostname kernel: RBP: 0000000000000004 R08: 000055cc2d2e22a0 R09: 00007fa9bd252d40
Nov 24 05:43:37 hostname kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 00007fa9af240000
Nov 24 05:43:37 hostname kernel: R13: 0000000000833419 R14: 000000000000000f R15: 0000000000000003
Nov 24 05:43:37 hostname kernel: Mem-Info:
Nov 24 05:43:37 hostname kernel: active_anon:17640 inactive_anon:182469 isolated_anon:0
active_file:812856 inactive_file:1292224 isolated_file:0
unevictable:0 dirty:6 writeback:0
slab_reclaimable:170338 slab_unreclaimable:66719
mapped:57643 shmem:41017 pagetables:4920 bounce:0
free:135449 free_pcp:0 free_cma:0
Nov 24 05:43:37 hostname kernel: Node 0 active_anon:70560kB inactive_anon:729876kB active_file:3251424kB inactive_file:5168896kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:230572kB dirty:24kB writeback:0kB shmem:164068kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 12288kB writeback_tmp:0kB kernel_stack:8400kB pagetables:19680kB all_unreclaimable? no
Nov 24 05:43:37 hostname kernel: Node 0 DMA free:13312kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 24 05:43:37 hostname kernel: lowmem_reserve[]: 0 2768 15745 15745 15745
Nov 24 05:43:37 hostname kernel: Node 0 DMA32 free:434312kB min:11972kB low:14964kB high:17956kB active_anon:7344kB inactive_anon:347200kB active_file:655228kB inactive_file:936252kB unevictable:0kB writepending:16kB present:3129152kB managed:2867008kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 24 05:43:37 hostname kernel: lowmem_reserve[]: 0 0 12977 12977 12977
Nov 24 05:43:37 hostname kernel: Node 0 Normal free:94172kB min:55544kB low:69428kB high:83312kB active_anon:63216kB inactive_anon:382852kB active_file:2596196kB inactive_file:4232836kB unevictable:0kB writepending:8kB present:13631488kB managed:13297412kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 24 05:43:37 hostname kernel: lowmem_reserve[]: 0 0 0 0 0
Nov 24 05:43:37 hostname kernel: Node 0 DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB (U) 2*2048kB (UM) 2*4096kB (M) = 13312kB
Nov 24 05:43:37 hostname kernel: Node 0 DMA32: 24038*4kB (UME) 22093*8kB (UME) 9551*16kB (UME) 293*32kB (UME) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 435088kB
Nov 24 05:43:37 hostname kernel: Node 0 Normal: 8267*4kB (UMEH) 1192*8kB (UMEH) 2403*16kB (UMEH) 391*32kB (UH) 10*64kB (H) 3*128kB (H) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 94588kB
Nov 24 05:43:37 hostname kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Nov 24 05:43:37 hostname kernel: Node 0 hugepages_total=2400 hugepages_free=287 hugepages_surp=0 hugepages_size=2048kB
Nov 24 05:43:37 hostname kernel: 2140592 total pagecache pages
Nov 24 05:43:37 hostname kernel: 65 pages in swap cache
Nov 24 05:43:37 hostname kernel: Swap cache stats: add 217664, delete 217599, find 49323/56024
Nov 24 05:43:37 hostname kernel: Free swap = 1385212kB
Nov 24 05:43:37 hostname kernel: Total swap = 2064380kB
Nov 24 05:43:37 hostname kernel: 4194157 pages RAM
Nov 24 05:43:37 hostname kernel: 0 pages HighMem/MovableOnly
Nov 24 05:43:37 hostname kernel: 149212 pages reserved
Nov 24 05:43:37 hostname kernel: 0 pages hwpoisoned
Nov 24 05:43:37 hostname kernel: SELinux: failed to load policy
Nov 24 05:43:38 hostname dnf-automatic[2350061]: SELinux: Could not load policy file /etc/selinux/targeted/policy/policy.31: Cannot allocate memory
Nov 24 05:43:38 hostname dnf-automatic[2350061]: load_policy: Can't load policy: Cannot allocate memory
Nov 24 05:43:38 hostname dbus-daemon[1100]: [system] Reloaded configuration
Nov 24 05:43:38 hostname dbus-daemon[1100]: [system] Reloaded configuration
Nov 24 05:43:38 hostname dbus-daemon[1100]: [system] Reloaded configuration
Nov 24 05:43:38 hostname dbus-daemon[1100]: [system] Reloaded configuration
Nov 24 05:43:40 hostname dbus-daemon[1100]: [system] Reloaded configuration
Nov 24 05:43:40 hostname dbus-daemon[1100]: [system] Reloaded configuration
Nov 24 05:43:41 hostname systemd-udevd[2351902]: Using default interface naming scheme 'rhel-8.0'.
Nov 24 05:43:42 hostname systemd[1]: Reloading.
Nov 24 05:43:43 hostname systemd[1]: Reloading.
Nov 24 05:43:47 hostname NetworkManager[1238]: <info> [1669265027.8825] manager: kernel firmware directory '/lib/firmware' changed
Due to this server being a database host, we have allocated hugepages and disabled overcommit. We have 2400 * 2048 kB memory allocated for hugepages and 11076044 kB available for userspace applications (CommitLimit). Total memory on the machine is, according to /proc/meminfo, 16179780 kB which should leave 188536 kB for the kernel. Perhaps this is not enough?
Here's our cat /proc/meminfo for one of the affected servers:
MemTotal: 16179780 kB
MemFree: 384992 kB
MemAvailable: 9381080 kB
Buffers: 3704 kB
Cached: 8809540 kB
SwapCached: 276 kB
Active: 3656528 kB
Inactive: 5692040 kB
Active(anon): 83920 kB
Inactive(anon): 611336 kB
Active(file): 3572608 kB
Inactive(file): 5080704 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 2064380 kB
SwapFree: 1387004 kB
Dirty: 92 kB
Writeback: 0 kB
AnonPages: 531136 kB
Mapped: 257248 kB
Shmem: 189552 kB
KReclaimable: 686400 kB
Slab: 953032 kB
SReclaimable: 686400 kB
SUnreclaim: 266632 kB
KernelStack: 8384 kB
PageTables: 19500 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 11076044 kB
Committed_AS: 2940032 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
Percpu: 101888 kB
HardwareCorrupted: 0 kB
AnonHugePages: 10240 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
FileHugePages: 0 kB
FilePmdMapped: 0 kB
HugePages_Total: 2400
HugePages_Free: 287
HugePages_Rsvd: 9
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 4915200 kB
DirectMap4k: 1275712 kB
DirectMap2M: 12355584 kB
DirectMap1G: 5242880 kB |
|
|
View Issue Details |
|
|
ID: |
Category: |
Severity: |
Reproducibility: |
Date Submitted: |
Last Update: |
|
662 |
[Rocky-Linux-8] NetworkManager |
major |
always |
2022-11-01 16:01 |
2022-11-07 22:12 |
|
|
Reporter: |
Carl Gromatzky |
Platform: |
ESXi 7.0.3, open-vm-tools 11.3.5 |
|
|
Assigned To: |
Louis Abel |
OS: |
Rocky 8.6 |
|
|
Priority: |
normal |
OS Version: |
4.18.0-372.9.1.e |
|
|
Status: |
needinfo |
Product Version: |
|
|
|
Product Build: |
|
Resolution: |
open |
|
|
Projection: |
none |
|
|
|
|
ETA: |
none |
Fixed in Version: |
|
|
|
|
|
Target Version: |
|
|
|
|
Summary: |
MAC Addr Corruption VMWare open-vm-tools 11.3.5-1.el8 |
|
Description: |
Subject:
Rock 8.6 VMXNET3 Device Enumeration Corruption
Message:
Running ESXi 7.0.3 build 20328353 (or build 20395099) on a UCS-C220-M5SX with firmware 4.2(2a) and a Intel Xeon Platinum 8168 2.7GHz processor
VM is Rocky 8.6 (4.18.0-372.9.1.el8.x86_64), using VMXNET3 network devices
Network devices mac addresses enumerate in the VM OS (Rocky 8.6) just fine, up to 3 network devices.
i.e.:
eth0 MAC addr = Network Adapter 1 MAC addr
eth1 MAC addr = Network Adapter 2 MAC addr
eth2 MAC addr = Network Adapter 3 MAC addr
Adding a fourth network device causes nmcli to show corrupted MAC address to device mappings. Rebooting causes dmesg errors and network loss (the driver appears to enumerate all vmxnet3 network devices incorrectly upon adding the 4th vmxnet3 device, mapping connection profiles to incorrect devices and causing kernel errors and network connectivity loss)
MAC addresses enumerate differently in the OS than the network adapters listed in vCenter after the fourth adapter is added to the vm,
i.e.:
eth0 MAC addr = Network Adapter 4 MAC addr
eth1 MAC addr = Network Adapter 1 MAC addr
eth2 MAC addr = Network Adapter 2 MAC addr
eth3 MAC addr = Network Adapter 3 MAC addr
open-vm-tools version experiencing this issue is 11.3.5-1.el8_6.1 from @AppStream repo
I also downloaded open-vm-tools versions 12.0.0, 12.0.5, 12.1.0 from the vmware github and rpm'd them into the build. All provide the same behavior on this vm. 12.1.0 actually started experiencing the MAC enumeration corruption when adding the third Network Adapter.
I opened a case with VMWare. They closed my case provided that I must open a case with Rocky Linux directly. |
|
Tags: |
|
|
Steps To Reproduce: |
The VM is given 1 vmxnet3 adapter to start. The network is configured and connectivity good.
The VM is given 1 additional adapter. The network connectivity is good.
The VM is given 1 additional adapter. The network connectivity is good.
The VM is given 1 additional adapter. The network connectivity fails, with mac address to nmcli connection mappings corrupted.
Generally, the error will not present until the VM is rebooted.
OR
The VM is given 4 vmxnet3 adapters to start. The network is configured and connectivity good.
The VM is rebooted, connectivity fails, with mac address to nmcli connection mappings corrupted. |
|
Additional Information: |
The OS is hardened, running in mult-user.target mode.
Running generic Rocky 8.6 build provides the following behavior in the same VMWare environment:
Add 4 vmxnet3 devices, the nmcli device list shows mac addr enumeration is fine, though the default network device nomenclature is ens[n].
I notice that the first 2 network devices enumerate as ens2[nn] and that the 3rd and 4th devices enumerate as ens1[nn].
In the production, hardened build, the network device nomenclature is eth[n]. |
|
Attached Files: |
netid_badrock.txt (1,771 bytes) 2022-11-04 21:29
https://bugs.rockylinux.org/file_download.php?file_id=331&type=bug
udevtest_badrock_afterreboot.txt (1,091 bytes) 2022-11-04 21:29
https://bugs.rockylinux.org/file_download.php?file_id=332&type=bug
udevtest_badrock.txt (1,096 bytes) 2022-11-04 21:29
https://bugs.rockylinux.org/file_download.php?file_id=333&type=bug
udevtest_r8default.txt (1,090 bytes) 2022-11-04 21:29
https://bugs.rockylinux.org/file_download.php?file_id=334&type=bug |
|
|
Notes |
(0000802)
Louis Abel
2022-11-01 16:08
|
Unfortunately we simply rebuild the open-vm-tools package as provided by Red Hat, which contains the sources that VMWare tools builds. So there's not much of a way for us to investigate.
I disagree with them that you must come to us to resolve this issue when it's their tools and platform. Please reopen a ticket and reference this bug ticket as well as https://bugs.rockylinux.org/view.php?id=200 to open a discussion with their engineering teams to find a solution. |
|
(0000803)
Carl Gromatzky
2022-11-01 16:15
|
I've update the VMWare ticket and am awaiting response from that support team, regarding the support-ability dispute. Thank you, Louis. |
|
(0000826)
Carl Gromatzky
2022-11-02 23:44
|
I uninstalled open-vm-tools 11.3.5 and confirmed in vcenter. Installed VMWare Tools 10.3.25, confirmed in vcenter, and experienced the same mac address corruption issue.
VMWare engineer updated. Perhaps, they will support VMWare Tools official. VMWare engineer also escalating as an open-vm-tools issue to VMWare global engineers (seeing if they will pickup at all).
The /sys/class/net/eth[0,1]/address sysfs files show the incorrectly mapped mac addresses. This seems to be a driver issue. Hoping to get feedback. Will update here. |
|
(0000827)
Louis Abel
2022-11-02 23:49
|
Thank you for the update. It's starting to sound like a platform issue (esx level). Perhaps ESX is providing the wrong information to the firmware, and the kernel/driver/udev is just following what it sees. Hoping to hear what they find.
One of our testers is currently moving their infra, so if vmware doesn't come back right away, he can probably try to see if the issue is reproducible in his environment with both Rocky Linux and RHEL. It would be interesting to see if it's also a problem in RHEL. |
|
(0000862)
Carl Gromatzky
2022-11-04 18:48
|
That's an interesting note. I researched and tested further, finding the VWMare Firmware/BIOS is indeed the culprit, in that it enumerates Network Adapters on the PCIe bus out of logical sequence. Red Hat referred to this as "giving the kernel information that doesn't make any sense," since 2016, so it's a long-ongoing issue. Below is text from the updated email (with additional research references that led to better understanding) that I sent to the VMWare engineer.
---------------------------------
To re-iterate the change in scope: I replicated the MAC Address to Interface corruption in the official VMWare Tools package. This is no longer only an open-vm-tools issue. This is a VMWare-specific problem, as noted by several vendors, and it has to due with PCI/PCIe bus enumeration problems of VMWare.
I believe that this VMWare article, which pulls information from Red Hat KBs, may prove the next path for testing:
https://docs.vmware.com/en/VMware-Adapter-for-SAP-Landscape-Management/services/Administration-Guide-for-LaMa-Administrators/GUID-0603A3F3-CCB1-42AF-A1E2-6B61979C00CB.html
I need to know more about how the PCIe Bus enumeration and Ethernet Port enumeration in VMWare function in order to create a reasonable udev rule to address VMWare's hardware handling.
Questions:
Is BDF bus assignment consistent and predictable when adding n number of Network Adapters to a VMWare VM?
What is the order of operations and resulting output of Bus ID assignment for n number of Network Adapters?
Are Ethernet port IDs user-definable in the Advanced Configuration Parameters of VM Settings?
I see that I can adjust advanced configuration port numbers for interfaces, but not sure if port number assignments would change bus ID behavior?
How are port numbers derived/calculated?
Would changing a port number persist a reboot, or would VMWare simply automatically reassign a port number?
Root cause of the issue:
Root cause of the MAC Addr corruption that takes Ethernet interfaces down appears to be that VMWare Tools passes bus IDs to the Rocky Linux kernel out of numerical order (this is a VMWare-specific issue, as noted by several OS vendors).
This behavior causes the naming sequence discrepancy for naming convention of Interface, ethX, to Ethernet Device Connection name, ensY.
The ethX old naming style is needed for many environments in order to comply with needs of devops and secops
Result:
Interface eth0 may map to Connection ens192 when Network Adapter 1 is first presented to the OS, then eth0 may map to ens161 on the next boot. This is due to ID_NET_NAME_PATH nomenclature assigned by udev, as enpMsN, enumerating out of logical sequence, as based on BUS ID enumeration from the VMWare Firmware/BIOS passing through to kernel udev/rules.d naming rules during POST.
i.e.
Interface eth0 will be enumerated first by the kernel and mapped to the first-presented virtual PCIe connection, which enumerates based on bus ID
Generall,y Network Adapter 1 maps to enp11s0 reliably
With 4 network adapters, the BUS ID doesn't matter at first presentation of the Network Adapter (hot-plugging gives Network Adapters 1-4 the appropriate eth0-3 names because they're being hot-plugged in order to the OS by VMWare)
On the next boot, Network Adapter 4 gets presented to the kernel on enp4s0, connection ens161, so udev then assignes eth0 to ens161 because the lowest numbers in the database get paired together
The MAC Address corruption grows in complexity with the number of VMWare Network Adapters added. If I give the VM 8 Network Adapters, the BUS IDs get injected in no predictable order, which goes back to the questions above. If I can understand how VMWare enumerates the PCIe bus and Ethernet ports, I believe I can create the udev rule to handle ethX naming at scale in el7+ (and other OS kernel) builds and satisfy dev and sec requirements.
Further links leading to above VMWare KB in research of this problem:
https://github.com/Azure/WALinuxAgent/issues/1750
https://github.com/coreos/bugs/issues/2437
https://github.com/systemd/systemd/pull/8458
https://unix.stackexchange.com/questions/611762/how-to-fix-the-conflict-in-the-naming-scheme-for-network-interfaces-use-by-predi
https://www.linuxsysadmins.com/systemd-network-interface-name-ensx-to-eth0/
https://www.redhat.com/en/blog/red-hat-enterprise-linux-73-achieving-persistent-and-consistent-network-interface-naming-vmware-environments
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/networking_guide/ch-consistent_network_device_naming#sec-Naming_Schemes_Hierarchy
https://communities.vmware.com/t5/vSphere-vNetwork-Discussions/how-to-fix-a-virtual-Network-Adapter-to-be-the-first-one-from/m-p/916181
https://kb.vmware.com/s/article/2047927
https://unix.stackexchange.com/questions/134483/why-is-my-ethernet-interface-called-enp0s10-instead-of-eth0
https://wiki.xenproject.org/wiki/Bus:Device.Function_(BDF)_Notation
https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/networking_guide/sec-understanding_the_predictable_network_interface_device_names
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_and_managing_networking/consistent-network-interface-device-naming_configuring-and-managing-networking
https://access.redhat.com/solutions/2592561 |
|
(0000863)
Carl Gromatzky
2022-11-04 21:29
|
I meant to attach some files for reference (see attached) |
|
(0000864)
Carl Gromatzky
2022-11-05 19:30
|
Keeping this string updated with research as discovered:
Based on the VMWare article and the Kemp article below, the community has understood the VMWare PCIe bus enumeration and configured a .vmx file that attaches max Ethernet devices and hard-codes their sequence to a logical order for the OS in user-space. This seems like a logical way to scale a template-able fix (which is my primary concern, as addressing this issue at scale any time someone wants more the 3 adapters would be inefficient). Any thoughts?
(I've ported this comment over to the VMWare engineer assigned. Want to make sure this passes VMWare's eyes and get the ok, regarding any pitfalls or obvious errors.)
Kemp article: https://support.kemptechnologies.com/hc/en-us/articles/201978745-When-adding-4-or-more-VMXNET3-NICs-to-a-VLM-in-VMware-the-order-is-incorrect
VMWare article: https://kb.vmware.com/s/article/2047927 |
|
(0000866)
Carl Gromatzky
2022-11-07 22:12
|
I deployed this config in a .vmx for a test vm, worked great
# 001.00101.00000 # PCI Slot=4 # Parent slot 21: # Bus:Dev.Func = 00h:15h.01h # Guest OS Order eth0
ethernet0.pciSlotNumber = "1184"
# 010.00101.00000 # PCI Slot=4 # Parent slot 21: # Bus:Dev.Func = 00h:15h.02h # Guest OS Order eth1
ethernet1.pciSlotNumber = "2208"
# 011.00101.00000 # PCI Slot=4 # Parent slot 21: # Bus:Dev.Func = 00h:15h.03h # Guest OS Order eth2
ethernet2.pciSlotNumber = "3232"
# 001.00110.00000 # PCI Slot=5 # Parent slot 22: # Bus:Dev.Func = 00h:16h.01h # Guest OS Order eth3
ethernet3.pciSlotNumber = "1216"
# 010.00110.00000 # PCI Slot=5 # Parent slot 22: # Bus:Dev.Func = 00h:16h.02h # Guest OS Order eth4
ethernet4.pciSlotNumber = "2240"
# 011.00110.00000 # PCI Slot=5 # Parent slot 22: # Bus:Dev.Func = 00h:16h.03h # Guest OS Order eth5
ethernet5.pciSlotNumber = "3264"
# 001.00111.00000 # PCI Slot=6 # Parent slot 23: # Bus:Dev.Func = 00h:17h.01h # Guest OS Order eth6
ethernet6.pciSlotNumber = "1248"
# 010.00111.00000 # PCI Slot=6 # Parent slot 23: # Bus:Dev.Func = 00h:17h.02h # Guest OS Order eth7
ethernet7.pciSlotNumber = "2272"
# 001.01000.00000 # PCI Slot=7 # Parent slot 24: # Bus:Dev.Func = 00h:18h.01h # Guest OS Order eth8
ethernet8.pciSlotNumber = "1280"
# 010.01000.00000 # PCI Slot=7 # Parent slot 24: # Bus:Dev.Func = 00h:18h.02h # Guest OS Order eth9
ethernet9.pciSlotNumber = "2304"
I also found that these .vmx lines need to be removed (they'll be automatically repopulated at next power state change of the vm)
ethernetN.dvs.portId = ...
ethernetN.generatedAddress = ...
ethernetN.addressType = ...
ethernetN.generateAddressOffset = ...
-------------------------------------------------------------------------------------------
If configuring a VM with the GUI, Need to Edit Settings > VM Optios (tab at the top) > Advanced > Edit Configuration > Add Configuration Params (particularly useful for new VM configuration)
Then, add ethernetN.pciSlotNumber and the slot ID noted from the lines above for the n-th interface (where N is the n-th interface in ethernetN)
-------------------------------------------------------------------------------------------
VMWare Engineer is also looking into opening a PR for VMWare-Tools and Rocky 8.6. (I would want to guess that this is a way to channel this issue into the right VMWare support workflow vs. VMWare assuming this an OS problem....the long years of history of this issue at the global level leaves interpretation up in the air, but I'm brave enough to feel hopeful lol) |
|
|
Notes |
(0000089)
Lukas Magauer
2021-11-16 15:31
|
Unfortunately got the output of a CentOS system on how it should look in the keyring in the earlier message, this is how it looks in RL 8.5:
keyctl show %:.platform
Keyring
874696905 ---lswrv 0 0 keyring: .platform
877754404 ---lswrv 0 0 \_ asymmetric: VMware, Inc.: 4ad8ba0472073d28127706ddc6ccb9050441bbc7
883063272 ---lswrv 0 0 \_ asymmetric: Microsoft Windows Production PCA 2011: a92902398e16c49778cd90f99e4f9ae17c55af53
1049370662 ---lswrv 0 0 \_ asymmetric: Microsoft Corporation UEFI CA 2011: 13adbf4309bd82709c8cd54f316ed522988a1bd4
459595990 ---lswrv 0 0 \_ asymmetric: VMware, Inc.: VMware Secure Boot Signing: 04597f3e1ffb240bba0ff0f05d5eb05f3e15f6d7
Furthermore, the CentOS OS type was used in the VM configuration as Rocky Linux is not available there. |
|
(0000090)
Sherif Nagy
2021-11-16 15:37
|
Also could you mention the hardware model underneath the ESXi? |
|
(0000091)
Lukas Magauer
2021-11-16 17:30
|
Off course!
My underlying system is a cluster of multiple Lenovo ThinkCentre M910q's with Intel Core i5-6500T,
EVC is enabled and uses the Intel Broadwell Generation CPU mode
Both DRS and HA are enabled
The keyprovider for TPM is also enabled
All VMs are using the VM hardware version 19 (ESXi 7U2 and later) |
|
(0000092)
Lukas Magauer
2021-11-19 10:39
|
After more investigation it looks to depend on the specific ESXi version as well, in my case it was 7U3a, but in chats it looks to also affect 7U2d. |
|
(0000093)
Sherif Nagy
2021-11-24 11:53
|
Looks like it is a shim bug: https://github.com/rhboot/shim/pull/387 and hopefully will be fixed when shim 15.5 released
Also someone else did test some versions of ESXi 7.0.2 and kdump was working fine with SB enabled, so not all ESXi version are affected by this. |
|
(0000094)
Alex Haydock
2021-12-20 14:36
|
I can reproduce this on VMware Workstation 15.1 too.
Looking at the upstream fix in PR #387 to the shim project on GitHub, it seems like the RCs are out in prep for a new release. RC2 was released 10 days ago, so hopefully it'll feed into RHEL too. |
|
(0000271)
Lukas Magauer
2022-07-16 12:55
|
As a small update, this also affects Rocky Linux 8.6, but got fixed for 9.0
Also seeing it for the latest versions of ESXi and Workstation. |
|
(0000310)
Lawrence Wright
2022-07-28 15:56
|
Are there any updates on when the Rocky shim-review for Shim 15.5 (or 15.6 for that matter) will be submitted? |
|
(0000311)
Sherif Nagy
2022-07-28 20:37
|
We did submit 15.5 awhile ago, but due to CVEs in shim 15.5 all reviews has been closed, most likely we will be waiting for the upstream to submit their 15.6 for review and then we will submit ours to make sure we are aligning with the upstream version |
|
(0000330)
Lawrence Wright
2022-08-09 10:40
|
It'd appear that upstream are now using 15.6, is Rocky able to move forward now? Thanks! |
|
(0000331)
Sherif Nagy
2022-08-09 13:29
|
They just pushed the sources to 15.6 but never got 15.6 reviewed or signed by Microsoft, it is basically still the same 15.5 signed shim from Microsoft, you can always check the review submissions here: https://github.com/rhboot/shim-review/issues , so I don't think the upstream got shim15.6 signed yet as per the review issues |
|
(0000332)
Lawrence Wright
2022-08-09 13:48
|
*heart sinks* yeah we've used your command above checked our upstream 8.6 box and can confirm same grep output as the above for shimx64.efi, this is with the package shim-x64-15.6-1.el8.x86_64.
Seems a bit... disappointing... for upstream to have a package versioned 15.6 actually containing a 15.5 shim.... |
|
(0000333)
Sherif Nagy
2022-08-09 14:41
|
Not disappointing, we had to do the same, to follow the upstream, that makes it easier so once the shim gets signed by MSFT, it can just be dropped into the shim pkg instead of changing lots of versions like shim-unsigned and such, but I agree, it is confusing :) |
|
(0000468)
Sherif Nagy
2022-08-24 10:05
|
We released shim-15.6 for rocky 8 and rocky 9, please update, test and provide us some feedback, other members in the community and testing team managed to test on their ESXi hardware and everything seems to be working fine. |
|
(0000698)
Lukas Magauer
2022-10-09 20:06
|
From my point of view, this is fixed and can be closed now! |
|
(0000700)
Sherif Nagy
2022-10-09 20:13
|
Perfect! thanks for testing |
|
(0000701)
Sherif Nagy
2022-10-09 20:13
|
Working fine with shim-15.6 |
|
|
Notes |
(0000529)
Louis Abel
2022-09-14 07:32
|
Thank you for the report.
What CPU is in your system? I would verify your CPU supports x86-64-v2. If your CPU supports x86-64-v2, it is recommended to pass your CPU to the virtual machine. You can do this by choosing "host" for your virtual machine CPU.
Setting to needinfo. |
|
(0000530)
MING-LI JIANG
2022-09-14 07:41
|
Linux icmnlab 5.15.53-1-pve #1 SMP PVE 5.15.53-1 (Fri, 26 Aug 2022 16:53:52 +0200) x86_64
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Wed Sep 14 15:34:49 CST 2022 on pts/0
root@icmnlab:~# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 46 bits physical, 48 bits virtual
CPU(s): 8
On-line CPU(s) list: 0-7
Thread(s) per core: 2
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) W-2123 CPU @ 3.60GHz
Stepping: 4
CPU MHz: 3600.000
CPU max MHz: 3900.0000
CPU min MHz: 1200.0000
BogoMIPS: 7200.00
Virtualization: VT-x
L1d cache: 128 KiB
L1i cache: 128 KiB
L2 cache: 4 MiB
L3 cache: 8.3 MiB
NUMA node0 CPU(s): 0-7
Vulnerability Itlb multihit: KVM: Mitigation: Split huge pages
Vulnerability L1tf: Mitigation; PTE Inversion; VMX EPT disabled
Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Meltdown: Mitigation; PTI
Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Retbleed: Mitigation; IBRS
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; IBRS, IBPB conditional, RSB filling
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT vulnerable
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1g
b rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_c
pl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand
lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriori
ty ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt
clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida ara
t pln pts hwp hwp_act_window hwp_epp hwp_pkg_req md_clear flush_l1d |
|
(0000531)
Louis Abel
2022-09-14 07:51
|
Please run: /lib64/ld-linux-x86-64.so.2 --help | grep x86-64-v2
"x86-64-v2 (supported, searched)" must return when you run this. In the event that it does, you will need to ensure your VM's are being passed "host" for the CPU. |
|
(0000532)
MING-LI JIANG
2022-09-14 08:01
|
|
|
(0000535)
Louis Abel
2022-09-15 02:42
|
Please read through the following link to help determine if your system supports x86-64-v2. https://unix.stackexchange.com/questions/631217/how-do-i-check-if-my-cpu-supports-x86-64-v2/631226 |
|
(0000665)
Joris Lambrecht
2022-10-06 06:00
|
Same problem.
CPU supports x86-64-v2
CPU supports x86-64-v3
What is the point of Rocky Linux refusing to work on kvm64 ?
Using host is wasteful in terms of resource usage.
Are there any boot flags which can be used to avoid this behavior ? |
|
(0000667)
Joris Lambrecht
2022-10-06 06:09
|
FYI/ https://bugzilla.redhat.com/show_bug.cgi?id=2060839 |
|
|
View Issue Details |
|
|
ID: |
Category: |
Severity: |
Reproducibility: |
Date Submitted: |
Last Update: |
|
299 |
[Rocky-Linux-8] selinux-policy |
minor |
N/A |
2022-09-15 19:44 |
2022-09-15 19:44 |
|
|
Reporter: |
Elliott Balsley |
Platform: |
|
|
|
Assigned To: |
|
OS: |
Rocky |
|
|
Priority: |
normal |
OS Version: |
8.6 |
|
|
Status: |
new |
Product Version: |
|
|
|
Product Build: |
|
Resolution: |
open |
|
|
Projection: |
none |
|
|
|
|
ETA: |
none |
Fixed in Version: |
|
|
|
|
|
Target Version: |
|
|
|
|
Summary: |
SELinux is preventing mdadm from ioctl access on the blk_file /dev/nvme11n1 |
|
Description: |
I'm seeing this SELinux message every few minutes. I have 22 identical NVMe drives in this machine, configured as RAID 60 using mdadm. I don't know what's unique about this one, but this is the only one showing this error. I believe mdadm should be allowed this access by default.
```
# sealert -l 917240ff-1abf-4e9f-980e-ae9400e137bb
SELinux is preventing /usr/sbin/mdadm from ioctl access on the blk_file /dev/nvme11n1.
***** Plugin catchall (100. confidence) suggests **************************
If you believe that mdadm should be allowed ioctl access on the nvme11n1 blk_file by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
Do
allow this access for now by executing:
# ausearch -c 'mdadm' --raw | audit2allow -M my-mdadm
# semodule -X 300 -i my-mdadm.pp
Additional Information:
Source Context system_u:system_r:pcp_pmcd_t:s0
Target Context system_u:object_r:nvme_device_t:s0
Target Objects /dev/nvme11n1 [ blk_file ]
Source mdadm
Source Path /usr/sbin/mdadm
Port <Unknown>
Host moonshine
Source RPM Packages
Target RPM Packages
SELinux Policy RPM selinux-policy-targeted-3.14.3-95.el8_6.4.noarch
Local Policy RPM selinux-policy-targeted-3.14.3-95.el8_6.4.noarch
Selinux Enabled True
Policy Type targeted
Enforcing Mode Permissive
Host Name moonshine
Platform Linux moonshine 4.18.0-372.19.1.el8_6.x86_64 #1
SMP Tue Aug 2 16:19:42 UTC 2022 x86_64 x86_64
Alert Count 944
First Seen 2022-09-07 00:03:11 PDT
Last Seen 2022-09-15 12:22:37 PDT
Local ID 917240ff-1abf-4e9f-980e-ae9400e137bb
Raw Audit Messages
type=AVC msg=audit(1663269757.936:7035): avc: denied { ioctl } for pid=339434 comm="mdadm" path="/dev/nvme11n1" dev="devtmpfs" ino=40986 ioctlcmd=0x1268 scontext=system_u:system_r:pcp_pmcd_t:s0 tcontext=system_u:object_r:nvme_device_t:s0 tclass=blk_file permissive=1
Hash: mdadm,pcp_pmcd_t,nvme_device_t,blk_file,ioctl
```
```
# ls -lZ /dev/nvme*n*
brw-rw----. 1 root disk system_u:object_r:nvme_device_t:s0 259, 1 Sep 7 09:25 /dev/nvme0n1
brw-rw----. 1 root disk system_u:object_r:nvme_device_t:s0 259, 7 Sep 7 09:25 /dev/nvme10n1
brw-rw----. 1 root disk system_u:object_r:nvme_device_t:s0 259, 9 Sep 7 09:25 /dev/nvme11n1
brw-rw----. 1 root disk system_u:object_r:nvme_device_t:s0 259, 10 Sep 7 09:25 /dev/nvme12n1
brw-rw----. 1 root disk system_u:object_r:nvme_device_t:s0 259, 12 Sep 7 09:25 /dev/nvme13n1
brw-rw----. 1 root disk system_u:object_r:nvme_device_t:s0 259, 23 Sep 7 09:25 /dev/nvme14n1
brw-rw----. 1 root disk system_u:object_r:nvme_device_t:s0 259, 15 Sep 7 09:25 /dev/nvme15n1
brw-rw----. 1 root disk system_u:object_r:nvme_device_t:s0 259, 21 Sep 7 09:25 /dev/nvme16n1
brw-rw----. 1 root disk system_u:object_r:nvme_device_t:s0 259, 13 Sep 7 09:25 /dev/nvme17n1
brw-rw----. 1 root disk system_u:object_r:nvme_device_t:s0 259, 24 Sep 7 09:25 /dev/nvme18n1
brw-rw----. 1 root disk system_u:object_r:nvme_device_t:s0 259, 20 Sep 7 09:25 /dev/nvme19n1
brw-rw----. 1 root disk system_u:object_r:nvme_device_t:s0 259, 0 Sep 7 09:25 /dev/nvme1n1
brw-rw----. 1 root disk system_u:object_r:nvme_device_t:s0 259, 29 Sep 7 09:25 /dev/nvme20n1
brw-rw----. 1 root disk system_u:object_r:nvme_device_t:s0 259, 28 Sep 7 09:25 /dev/nvme21n1
brw-rw----. 1 root disk system_u:object_r:nvme_device_t:s0 259, 11 Sep 7 09:25 /dev/nvme22n1
brw-rw----. 1 root disk system_u:object_r:nvme_device_t:s0 259, 16 Sep 7 09:25 /dev/nvme22n1p1
brw-rw----. 1 root disk system_u:object_r:nvme_device_t:s0 259, 17 Sep 7 09:25 /dev/nvme22n1p2
brw-rw----. 1 root disk system_u:object_r:nvme_device_t:s0 259, 19 Sep 7 09:25 /dev/nvme22n1p3
brw-rw----. 1 root disk system_u:object_r:nvme_device_t:s0 259, 22 Sep 7 09:25 /dev/nvme23n1
brw-rw----. 1 root disk system_u:object_r:nvme_device_t:s0 259, 25 Sep 7 09:25 /dev/nvme23n1p1
brw-rw----. 1 root disk system_u:object_r:nvme_device_t:s0 259, 26 Sep 7 09:25 /dev/nvme23n1p2
brw-rw----. 1 root disk system_u:object_r:nvme_device_t:s0 259, 27 Sep 7 09:25 /dev/nvme23n1p3
brw-rw----. 1 root disk system_u:object_r:nvme_device_t:s0 259, 2 Sep 7 09:25 /dev/nvme2n1
brw-rw----. 1 root disk system_u:object_r:nvme_device_t:s0 259, 3 Sep 7 09:25 /dev/nvme3n1
brw-rw----. 1 root disk system_u:object_r:nvme_device_t:s0 259, 4 Sep 7 09:25 /dev/nvme4n1
brw-rw----. 1 root disk system_u:object_r:nvme_device_t:s0 259, 5 Sep 7 09:25 /dev/nvme5n1
brw-rw----. 1 root disk system_u:object_r:nvme_device_t:s0 259, 6 Sep 7 09:25 /dev/nvme6n1
brw-rw----. 1 root disk system_u:object_r:nvme_device_t:s0 259, 8 Sep 7 09:25 /dev/nvme7n1
brw-rw----. 1 root disk system_u:object_r:nvme_device_t:s0 259, 18 Sep 7 09:25 /dev/nvme8n1
brw-rw----. 1 root disk system_u:object_r:nvme_device_t:s0 259, 14 Sep 7 09:25 /dev/nvme9n1
```
```
# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sr0 11:0 1 2.1G 0 rom
nvme1n1 259:0 0 5.8T 0 disk
└─md101 9:101 0 52.4T 0 raid6
└─md0 9:0 0 104.8T 0 raid0
└─md0p1 259:30 0 104.8T 0 md /mnt/raid
nvme0n1 259:1 0 5.8T 0 disk
└─md101 9:101 0 52.4T 0 raid6
└─md0 9:0 0 104.8T 0 raid0
└─md0p1 259:30 0 104.8T 0 md /mnt/raid
nvme2n1 259:2 0 5.8T 0 disk
└─md101 9:101 0 52.4T 0 raid6
└─md0 9:0 0 104.8T 0 raid0
└─md0p1 259:30 0 104.8T 0 md /mnt/raid
nvme3n1 259:3 0 5.8T 0 disk
└─md101 9:101 0 52.4T 0 raid6
└─md0 9:0 0 104.8T 0 raid0
└─md0p1 259:30 0 104.8T 0 md /mnt/raid
nvme4n1 259:4 0 5.8T 0 disk
└─md101 9:101 0 52.4T 0 raid6
└─md0 9:0 0 104.8T 0 raid0
└─md0p1 259:30 0 104.8T 0 md /mnt/raid
nvme5n1 259:5 0 5.8T 0 disk
└─md101 9:101 0 52.4T 0 raid6
└─md0 9:0 0 104.8T 0 raid0
└─md0p1 259:30 0 104.8T 0 md /mnt/raid
nvme6n1 259:6 0 5.8T 0 disk
└─md101 9:101 0 52.4T 0 raid6
└─md0 9:0 0 104.8T 0 raid0
└─md0p1 259:30 0 104.8T 0 md /mnt/raid
nvme10n1 259:7 0 5.8T 0 disk
└─md101 9:101 0 52.4T 0 raid6
└─md0 9:0 0 104.8T 0 raid0
└─md0p1 259:30 0 104.8T 0 md /mnt/raid
nvme7n1 259:8 0 5.8T 0 disk
└─md101 9:101 0 52.4T 0 raid6
└─md0 9:0 0 104.8T 0 raid0
└─md0p1 259:30 0 104.8T 0 md /mnt/raid
nvme11n1 259:9 0 5.8T 0 disk
└─md102 9:102 0 52.4T 0 raid6
└─md0 9:0 0 104.8T 0 raid0
└─md0p1 259:30 0 104.8T 0 md /mnt/raid
nvme12n1 259:10 0 5.8T 0 disk
└─md102 9:102 0 52.4T 0 raid6
└─md0 9:0 0 104.8T 0 raid0
└─md0p1 259:30 0 104.8T 0 md /mnt/raid
nvme22n1 259:11 0 477G 0 disk
├─nvme22n1p1 259:16 0 1M 0 part
├─nvme22n1p2 259:17 0 1.5G 0 part
└─nvme22n1p3 259:19 0 475.4G 0 part
└─ubuntu--vg-ubuntu--lv 253:3 0 100G 0 lvm
nvme13n1 259:12 0 5.8T 0 disk
└─md102 9:102 0 52.4T 0 raid6
└─md0 9:0 0 104.8T 0 raid0
└─md0p1 259:30 0 104.8T 0 md /mnt/raid
nvme17n1 259:13 0 5.8T 0 disk
└─md102 9:102 0 52.4T 0 raid6
└─md0 9:0 0 104.8T 0 raid0
└─md0p1 259:30 0 104.8T 0 md /mnt/raid
nvme9n1 259:14 0 5.8T 0 disk
└─md101 9:101 0 52.4T 0 raid6
└─md0 9:0 0 104.8T 0 raid0
└─md0p1 259:30 0 104.8T 0 md /mnt/raid
nvme15n1 259:15 0 5.8T 0 disk
└─md102 9:102 0 52.4T 0 raid6
└─md0 9:0 0 104.8T 0 raid0
└─md0p1 259:30 0 104.8T 0 md /mnt/raid
nvme8n1 259:18 0 5.8T 0 disk
└─md101 9:101 0 52.4T 0 raid6
└─md0 9:0 0 104.8T 0 raid0
└─md0p1 259:30 0 104.8T 0 md /mnt/raid
nvme19n1 259:20 0 5.8T 0 disk
└─md102 9:102 0 52.4T 0 raid6
└─md0 9:0 0 104.8T 0 raid0
└─md0p1 259:30 0 104.8T 0 md /mnt/raid
nvme16n1 259:21 0 5.8T 0 disk
└─md102 9:102 0 52.4T 0 raid6
└─md0 9:0 0 104.8T 0 raid0
└─md0p1 259:30 0 104.8T 0 md /mnt/raid
nvme23n1 259:22 0 477G 0 disk
├─nvme23n1p1 259:25 0 600M 0 part /boot/efi
├─nvme23n1p2 259:26 0 1G 0 part /boot
└─nvme23n1p3 259:27 0 475.4G 0 part
├─rl-root 253:0 0 70G 0 lvm /
├─rl-swap 253:1 0 4G 0 lvm [SWAP]
└─rl-home 253:2 0 401.4G 0 lvm /home
nvme14n1 259:23 0 5.8T 0 disk
└─md102 9:102 0 52.4T 0 raid6
└─md0 9:0 0 104.8T 0 raid0
└─md0p1 259:30 0 104.8T 0 md /mnt/raid
nvme18n1 259:24 0 5.8T 0 disk
└─md102 9:102 0 52.4T 0 raid6
└─md0 9:0 0 104.8T 0 raid0
└─md0p1 259:30 0 104.8T 0 md /mnt/raid
nvme21n1 259:28 0 5.8T 0 disk
└─md102 9:102 0 52.4T 0 raid6
└─md0 9:0 0 104.8T 0 raid0
└─md0p1 259:30 0 104.8T 0 md /mnt/raid
nvme20n1 259:29 0 5.8T 0 disk
└─md102 9:102 0 52.4T 0 raid6
└─md0 9:0 0 104.8T 0 raid0
└─md0p1 259:30 0 104.8T 0 md /mnt/raid
``` |
|
Tags: |
|
|
Steps To Reproduce: |
|
|
Additional Information: |
|
|
Attached Files: |
|
|
|
Notes |
(0000170)
Neil Hanlon
2022-05-17 03:53
|
Thank you for the report. You're right the image seems not very functional. I did basic smoke tests on it, but clearly need to validate some more.
I'll rebuild this in the morning and check what's going on. |
|
(0000176)
Martin Nix
2022-05-20 11:59
|
Also of note here is that the cloud image is missing LVM support at boot for the root filesystem by default (I know that is probably not minimal but in my estimation it is essential for most instances) |
|
(0000180)
Ian Walker
2022-05-23 09:45
|
The majority of cloud images do not have LVM in them, at least as far as I have seen with all downloaded images over the past few years that I've used with Openstack. Usually an image has a single standard partition, since cloud-init manages the images, and if an instance is resized, then it uses growpart to increase the disk partition. If the image has an LVM partition, then chances are this isn't going to work. |
|
(0000182)
Martin Nix
2022-05-24 10:33
|
Thanks Ian - but I've found the opposite to be true.
Our use case is the same in Openstack, we have been using Centos images (from the Centos qcow resource at https://cloud.centos.org/centos/7/images) and more recently the Rocky image in question. From 7.2 right up to and including 8.5 LVM for the root file system was enabled by default and works completely as expected with growpart. The problem we have now with this image is partly to do with growpart failing due specifically due to LVM being missing (makes this image a special case to work around for our build system)
For now I've taken the 8.5 image (which works absolutely fine) and updated it to 8.6 using virt tools (as a workaround) |
|
(0000183)
Ian Walker
2022-05-24 10:49
|
Martin, OK good to know :) I know previously a lot of the centos/rhel/sles jeos images that I used were standard images, hadn't seen anything with lvm yet, so if that's the case, fair enough :) |
|
(0000184)
TREVOR BENSON
2022-05-25 00:00
|
@marnixgb I've seem the same as you in most cases.
@iwalker Which image are you referring to? The latest x86_64 image at https://cloud.centos.org/centos/7/images, **CentOS-7-x86_64-GenericCloud-2111.qcow2** does not have any `lvm2` RPMs installed by default. Neither does the **rhel-8.5-x86_64-kvm_password.qcow2** image available from redhat.com. I have not used the ppc64le or the aarch64 images, so am not sure they handle disks the same way.
Are you sure the image wasn't customized first, or a cloud-init configuration is installing them for you? |
|
(0000185)
TREVOR BENSON
2022-05-25 00:02
|
You can ignore the _password on the qcow image. I just happened to use virt-customize really quickly to set the password of root for a quick confirmation test without modifying the original image. |
|
(0000186)
Ian Walker
2022-05-25 07:01
|
@Trevor Benson
I used the official images downloaded from CentOS, RHEL, Debian, Ubuntu. To date I didn't see these images with LVM installed, let alone with LVM partitions (but they were images pre December 2021 - not downloaded new ones since). But if things change and they start to use LVM like @Martin said then fair enough. I only mentioned, because if the majority of other images do not have it, then Rocky can also be without LVM as well. It doesn't make a huge difference, since growpart will resize it anyway. |
|
(0000187)
Steve Brasier
2022-05-25 13:24
|
Also doesn't appear to have `hostname` installed. |
|
(0000188)
TREVOR BENSON
2022-05-25 15:26
|
@iwalker I got the @'s backwards in my message. I was intending to @ you that I saw the same as you (no lvm) in many images from upstream. Sorry for the confusion by swapping the @'s in the note. |
|
(0000189)
Martin Nix
2022-05-26 08:00
|
First off apologies on the LVM front - I saw something and jumped to a conclusion (the wrong one) that was incorrect as it turns out :
On images of 8.5 and prior the filesystem is created by anaconda as below :
[root@test85 ~]# lsblk -f
NAME FSTYPE LABEL UUID MOUNTPOINT
vda
└─vda1 xfs 4c90d825-f649-4c1b-97b8-11f283331ef5 /
vdb swap 50020188-cb9d-4df7-8e0e-2391886d4978 [SWAP]
[root@test85 ~]# cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Sun Nov 14 18:50:49 2021
#
# Accessible filesystems, by reference, are maintained under '/dev/disk/'.
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info.
#
# After editing this file, run 'systemctl daemon-reload' to update systemd
# units generated from this file.
#
UUID=4c90d825-f649-4c1b-97b8-11f283331ef5 / xfs defaults 0 0
/dev/vdb none swap sw,comment=cloudconfig 0 0
On the 8.6 image the filesystem is as below (quite different) :
[root@test86 ~]# lsblk -f
NAME FSTYPE LABEL UUID MOUNTPOINT
vda
├─vda1 vfat EFI 49AB-A4B4
├─vda2
└─vda3 xfs root ebb44e1a-2801-4e1e-a8c5-921204153736 /
vdb swap b2e2ed61-7ca0-4c82-a2e7-fa945ec59adc [SWAP]
[root@test86 ~]# cat /etc/fstab
/dev/vdb none swap sw,comment=cloudconfig 0 0
I have bought up two test machines, one 8.5 and one 8.6, from the stock qcow2 images and got together the below list - these are the packages that were in 8.5 but are no longer in 8.6. There are no packages in 8.6 that did not exist in 8.5.
abattis-cantarell-fonts.noarch
audit.x86_64
authselect-compat.x86_64
authselect-libs.x86_64
authselect.x86_64
cairo-gobject.x86_64
cairo.x86_64
c-ares.x86_64
cronie-anacron.x86_64
cronie.x86_64
crontabs.noarch
crypto-policies-scripts.noarch
dejavu-fonts-common.noarch
dejavu-sans-mono-fonts.noarch
dmidecode.x86_64
dracut-config-rescue.x86_64
elfutils-debuginfod-client.x86_64
fontconfig.x86_64
fontpackages-filesystem.noarch
gdk-pixbuf2.x86_64
geolite2-city.noarch
geolite2-country.noarch
gnupg2-smime.x86_64
gobject-introspection.x86_64
groff-base.x86_64
grubby.x86_64
hardlink.x86_64
hdparm.x86_64
hostname.x86_64
hwdata.noarch
initscripts.x86_64
irqbalance.x86_64
kbd-legacy.noarch
kbd-misc.noarch
kbd.x86_64
kernel-tools-libs.x86_64
kernel-tools.x86_64
kpartx.x86_64
less.x86_64
libappstream-glib.x86_64
libdaemon.x86_64
libdhash.x86_64
libestr.x86_64
libfastjson.x86_64
libldb.x86_64
libmaxminddb.x86_64
libnl3-cli.x86_64
libnl3.x86_64
libpipeline.x86_64
libsecret.x86_64
libsoup.x86_64
libsss_autofs.x86_64
libsss_certmap.x86_64
libsss_idmap.x86_64
libsss_nss_idmap.x86_64
libsss_sudo.x86_64
libstemmer.x86_64
libsysfs.x86_64
libtalloc.x86_64
libtdb.x86_64
libteam.x86_64
libtevent.x86_64
libuser.x86_64
libX11-common.noarch
libX11.x86_64
libXau.x86_64
libxcb.x86_64
libXext.x86_64
libxkbcommon.x86_64
libXrender.x86_64
lmdb-libs.x86_64
logrotate.x86_64
lshw.x86_64
lsscsi.x86_64
man-db.x86_64
memstrack.x86_64
microcode_ctl.x86_64
mozjs60.x86_64
NetworkManager-team.x86_64
NetworkManager-tui.x86_64
newt.x86_64
numactl-libs.x86_64
oddjob-mkhomedir.x86_64
oddjob.x86_64
openssl-pkcs11.x86_64
PackageKit-glib.x86_64
PackageKit.x86_64
parted.x86_64
passwd.x86_64
pciutils-libs.x86_64
pigz.x86_64
pinentry.x86_64
pixman.x86_64
platform-python-pip.noarch
policycoreutils-python-utils.noarch
polkit-pkla-compat.x86_64
polkit.x86_64
prefixdevname.x86_64
python3-cairo.x86_64
python3-decorator.noarch
python3-gobject-base.x86_64
python3-gobject.x86_64
python3-linux-procfs.noarch
python3-netifaces.x86_64
python3-perf.x86_64
python3-pexpect.noarch
python3-ptyprocess.noarch
python3-pydbus.noarch
python3-pyudev.noarch
python3-slip-dbus.noarch
python3-slip.noarch
python3-syspurpose.x86_64
python3-systemd.x86_64
python3-unbound.x86_64
rocky-logos.x86_64
rpm-plugin-selinux.x86_64
rpm-plugin-systemd-inhibit.x86_64
rsyslog.x86_64
selinux-policy.noarch
selinux-policy-targeted.noarch
setroubleshoot-plugins.noarch
setroubleshoot-server.x86_64
sg3_utils-libs.x86_64
sg3_utils.x86_64
shared-mime-info.x86_64
slang.x86_64
sscg.x86_64
sssd-client.x86_64
sssd-common.x86_64
sssd-kcm.x86_64
sssd-nfs-idmap.x86_64
sudo.x86_64
teamd.x86_64
timedatex.x86_64
trousers-lib.x86_64
trousers.x86_64
tuned.noarch
unbound-libs.x86_64
virt-what.x86_64
xkeyboard-config.noarch |
|
(0000190)
Ian Walker
2022-05-26 16:07
|
Seems strange, normally images are a single disk, and yet this mentioned vda for the main partition and vdb for swap (unless that was added later and not part of the initial Rocky image).
As with all the other images I've mentioned, they were all single disks and partitions, no swap, so I think the Rocky image doesn't need this second disk or swap for that matter, it can only be more problematic this way. Generally, if I spin up an instance on Openstack or wherever (eg: other vps hosting platforms that spin up from an image), it's easy enough to add swap using fallocate and dropping the file on the main disk partition. |
|
(0000191)
Martin Nix
2022-05-27 07:22
|
You can safely ignore the vdb entries - they are a product of bringing up instances in openstack where the swap is defined in the flavor (which we do as standard through TerraForm)
I was trying to illustrate that the regular partitioning standard that occurs at init typically is not happening in this case (with 8.6) |
|
(0000229)
Jere Kataja
2022-06-03 18:05
|
Also to note that static IP assignment with generic cloud image fails with 8.6. With 8.5 it worked normally. Might be related to the same deficiencies listed above. |
|
(0000230)
Neil Hanlon
2022-06-03 18:37
|
Hi all,
Thank you for the reports here and for your patience. I've got some new images cooking and should have them up later tonight!
The short resolution is, as I should have known, "If it ain't broke, don't fix it" (: |
|
(0000231)
Juha Rajamaki
2022-06-04 03:53
|
Please consider also one more thing in the new image creation. It is that "virsh console" does not work to a VM running 8.6 image on kvm hypervisor. I think it is because it is missing "console=ttyS0" in the kernel command line, but has "rhgb" which means graphical console.
The console of 8.6 works when viewed through the graphical console of OpenStack but for most users it is important that "virsh console" would work as it has always been in earlier images.
Rocky 8.5:
$ dmesg |grep -i boot_image
[ 0.000000] Command line: BOOT_IMAGE=(hd0,msdos1)/boot/vmlinuz-4.18.0-348.el8.0.2.x86_64 root=UUID=4c90d825-f649-4c1b-97b8-11f283331ef5 ro console=ttyS0,115200n8 no_timer_check net.ifnames=0 crashkernel=auto
Rocky 8.6:
$ dmesg |grep -i boot_image
[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.18.0-372.9.1.el8.x86_64 root=UUID=ebb44e1a-2801-4e1e-a8c5-921204153736 ro rhgb selinux=0 audit=0 rw |
|
(0000233)
Jere Kataja
2022-06-07 06:40
|
Apologies for pestering Neil but any guesstimate for when you would have new images available to download? :-) |
|
(0000250)
Jere Kataja
2022-07-07 06:04
|
Noticed that there is a new image available (20220702.0) and in our tests the x86_64 version at least appears to be working as expected:
- Static IP assignment working
- virsh console access working
- Ansible deployment in general (included sudo etc.) working
Thanks! |
|
(0000252)
David Monro
2022-07-08 00:13
|
Looks good to me, deploys just like 8.5 through our ansible/openstack (which the original image did not).
Cheers
David |
|
(0000253)
Neil Hanlon
2022-07-08 00:40
|
Excellent news.
Thank you everyone for your extreme patience with getting this fixed. In the future this will not be a problem. |
|
(0000254)
Martin Nix
2022-07-08 08:14
|
Looks good but I would just like to add - the image really needs virt-sparsify applied
The original image is 2674917376, applying a sparsify myself brings that down to 1402798080 - that's a pretty epic difference |
|
(0000255)
Neil Hanlon
2022-07-11 01:59
|
Thanks Martin -
I've compressed them by default. The new ones will come out sometime this week, pre-compressed.
https://git.resf.org/sig_core/toolkit/commit/1f9468092453d9025fe57b570c615617f1904f0e |
|
(0000315)
Steve Brasier
2022-08-02 13:09
|
Any progress on sparsified images please? https://download.rockylinux.org/pub/rocky/8/images/ only seems to have the `..0702` images still. |
|
|
View Issue Details |
|
|
ID: |
Category: |
Severity: |
Reproducibility: |
Date Submitted: |
Last Update: |
|
163 |
[Rocky-Linux-8] NetworkManager |
minor |
always |
2022-07-26 18:32 |
2022-07-26 18:32 |
|
|
Reporter: |
Tim Orbaker |
Platform: |
x64 (Inspiron 3783) |
|
|
Assigned To: |
|
OS: |
Rocky Linux |
|
|
Priority: |
none |
OS Version: |
8.5 |
|
|
Status: |
new |
Product Version: |
|
|
|
Product Build: |
|
Resolution: |
open |
|
|
Projection: |
none |
|
|
|
|
ETA: |
none |
Fixed in Version: |
|
|
|
|
|
Target Version: |
|
|
|
|
Summary: |
Problems with (QC9377) ath10k partially disconnecting with firmware api6 |
|
Description: |
After an amount of time without establishing a new connection, the wireless network interface (Qualcomm Atheros QCA9377 802.11ac Wireless Network Adapter (rev 31)) refuses to connect to anything until the wifi is disabled and restarted. How it is restarted (gnome interface, restarting NetworkManager or rebooting) does not seem to matter in terms of whether or not function returns or how long the interval is between occurrences.
The wired interface is not affected when this happens.
Interestingly, the interface does not have to be idle, and existing connections are not terminated (Zoom meetings and SSH sessions), but new connections cannot be created (ping, DNS queries, new SSH sessions). I have continued a Zoom meeting long after my browser stopped being able to do DNS resolution without an issue.
No messages were found in any of the logs or with dmesg when this occurs, and the network requests for new connections eventually time out. The interface still shows that it is connected and the system acts as if the Internet is disconnected at the router, although existing connections are not affected.
Disabling power management, first for the QC9377 network interface, and then for the entire system did not produce any noticable difference. it happens at roughly the same frequency whether connected to power or not.
All available updates were applied.
The problem turned out to be the firmware file: /lib/firmware/ath10k/QCA9377/hw1.0/firmware-6.bin
If this is removed, then /lib/firmware/ath10k/QCA9377/hw1.0/firmware-5.bin is used and the system functions properly. To verify the fix, use the desg command below and look for "api 5"
System information: This is a Dell Inspiron 3783
Diagnosis: see line 5 below ("api 6")
$ dmesg | grep '] ath10k'
[ 6.623522] ath10k_pci 0000:03:00.0: enabling device (0000 -> 0002)
[ 6.624123] ath10k_pci 0000:03:00.0: pci irq msi oper_irq_mode 2 irq_mode 0 reset_mode 0
[ 6.830599] ath10k_pci 0000:03:00.0: qca9377 hw1.1 target 0x05020001 chip_id 0x003821ff sub 1028:1810
[ 6.830604] ath10k_pci 0000:03:00.0: kconfig debug 0 debugfs 1 tracing 0 dfs 0 testmode 0
[ 6.831134] ath10k_pci 0000:03:00.0: firmware ver WLAN.TF.1.0-00002-QCATFSWPZ-5 api 6 features ignore-otp crc32 c3e0d04f
[ 6.894966] ath10k_pci 0000:03:00.0: board_file api 2 bmi_id N/A crc32 8aedfa4a
[ 6.996282] ath10k_pci 0000:03:00.0: htt-ver 3.44 wmi-op 4 htt-op 3 cal otp max-sta 32 raw 0 hwcrypto 1
[ 7.070153] ath10k_pci 0000:03:00.0 wlp3s0: renamed from wlan0
Device information
$ sudo lspci -vvvvv
03:00.0 Network controller: Qualcomm Atheros QCA9377 802.11ac Wireless Network Adapter (rev 31)
Subsystem: Dell Device 1810
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 142
Region 0: Memory at b2000000 (64-bit, non-prefetchable) [size=2M]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable+ Count=1/8 Maskable+ 64bit-
Address: fee004d8 Data: 0000
Masking: 000000fe Pending: 00000000
Capabilities: [70] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 10.000W
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr+ TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <4us, L1 <64us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk+
ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s (ok), Width x1 (ok)
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Not Supported, TimeoutDis+ NROPrPrP- LTR+
10BitTagComp- 10BitTagReq- OBFF Via message, ExtFmt- EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS- TPHComp- ExtTPHComp-
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR+ OBFF Disabled,
AtomicOpsCtl: ReqEn-
LnkCap2: Supported Link Speeds: 2.5GT/s, Crosslink- Retimer- 2Retimers- DRS-
LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr+ BadTLP+ BadDLLP+ Rollover- Timeout+ AdvNonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
AERCap: First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap- ECRCChkEn-
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 00000000 00000000 00000000 00000000
Capabilities: [148 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [168 v1] Device Serial Number 00-00-00-00-00-00-00-00
Capabilities: [178 v1] Latency Tolerance Reporting
Max snoop latency: 3145728ns
Max no snoop latency: 3145728ns
Capabilities: [180 v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
PortCommonModeRestoreTime=50us PortTPowerOnTime=10us
L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
T_CommonMode=0us LTR1.2_Threshold=81920ns
L1SubCtl2: T_PwrOn=10us
Kernel driver in use: ath10k_pci
Kernel modules: ath10k_pci
|
|
Tags: |
glitch |
|
Steps To Reproduce: |
The amount of time to the appearance of the problem is consistent. The appearance of the problem is consistent. The key feature is going a period of time without establishing a new connection. Allowing the system to sit idle will cause the problem as well.
The system may be idle, but this is not a condition of the problem. I have had it happen during a ZOOM meeting, I have had it happen during an SSH session. What is consistent is that the wireless network interface refuses to establish any new connections, even to local addresses.
|
|
Additional Information: |
I am unsure if this the proper place to report this (it is an upstream product, I can't find a section for wireless drivers or the ath10k driver).
It drove me crazy for a week and a half I just want anyone else having the problem to be able to find the solution somewhere.
|
|
Attached Files: |
|
|
|
Notes |
(0000096)
Alex Corcoles
2022-02-03 10:02
|
To be clear, I think the bug is the mtime on /proc/1 on a Raspberry Pi. On other systems it's apparently the boot time. |
|
(0000113)
Skip Grube
2022-05-03 14:46
|
Just adding an update here: I'm 99.9% certain this is because the Raspberry Pi has no internal clock, and so it defaults to Jan 1, 1970 (epoch) when the system does its early boot.
I'm still pondering this issue, but I don't really have a good solution right now. Best bet as a workaround (for now) is to patch that DNF plugin to use a different source for system boot time.
I would expect this same behavior on any system with no internal clock, a dead CMOS battery, etc.
Still brainstorming if there's a better solution to this. |
|
(0000115)
Alex Corcoles
2022-05-03 16:48
|
Yeah, I had no idea that the RPI has no clock :( Everything fits that explanation, IMHO.
My only concern is that I don't think RHEL 8 supports any hardware with a "malfunctioning" clock, so they might be hesitant to patch the DNF plugin to do the proper thing (although I *think* that's the most correct solution). I am happy to file a bug (to the CentOS 8 bug tracker, corresponding to the DNF plugin RPM, I assume?) and see how that plays out. (If you have a more direct communication with upstream, maybe you want to report that yourself?)
Thinking out a little bit, the other options I can think of:
* Do something similar to fake-hwclock. However, fake-hwclock specifically seems to be a systemd service, so I think it would be too late for this to work. And I didn't see anything like a kernel command-line parameter that lets you set a custom time.
* Find a way to change the mtime of /proc/1 when the date is set via NTP. I only thought to remount /proc, but that doesn't seem to change anything
* Create an alternate package just for Rocky Linux 8 / RPI that overrides the DNF plugin. I think this option is worse than all other proposals.
Let me know your thoughts. If I don't hear anything, I'll file a bug to CentOS 8 Stream python3-dnf-plugins-core in 1-2 days.
Cheers,
Álex |
|
(0000116)
Alex Corcoles
2022-05-05 18:19
|
I filed https://bugzilla.redhat.com/show_bug.cgi?id=2082295 , following the instructions at https://github.com/rpm-software-management/dnf-plugins-core#bug-reporting-etc . |
|
|
Notes |
(0000074)
Louis Abel
2021-06-12 09:35
|
Hello.
According to some of the messages, they are leading to believe that either A) there is a kernel or module bug or more likely B) the backend storage device is having issues (whether disk is failing, cabling, etc). This is based on the following logs (ignoring the call obvious traces):
Jun 12 01:01:02 rocky8 kernel: XFS (dm-0): metadata I/O error in xfs_da_read_buf+0xcf/0x120 [xfs] at daddr 0x29b0a48 len 8 error 5
...
Jun 12 01:00:17 rocky8 kernel: virtio_scsi virtio1: request:id 549 is not a head!
Jun 12 01:00:17 rocky8 kernel: sd 0:0:0:0: [sda] tag#958 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
Jun 12 01:00:17 rocky8 kernel: sd 0:0:0:0: [sda] tag#958 CDB: Write(10) 2a 00 01 76 38 d0 00 00 18 00
Jun 12 01:00:17 rocky8 kernel: blk_update_request: I/O error, dev sda, sector 24525008 op 0x1:(WRITE) flags 0x800 phys_seg 3 prio class 0
...
Jun 12 01:18:36 rocky8 kernel: dm-0: writeback error on inode 52098861, offset 2265088, sector 45857848
Jun 12 01:18:36 rocky8 kernel: sd 0:0:0:0: [sda] tag#903 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
Jun 12 01:18:36 rocky8 kernel: sd 0:0:0:0: [sda] tag#903 CDB: Write(10) 2a 00 03 42 4c d8 00 00 08 00
Jun 12 01:18:36 rocky8 kernel: blk_update_request: I/O error, dev sda, sector 54676696 op 0x1:(WRITE) flags 0x100000 phys_seg 1 prio class 0
...
Jun 12 01:20:04 rocky8 kernel: Read-error on swap-device (253:1:93232)
Jun 12 01:27:54 rocky8 kernel: sd 0:0:0:0: [sda] tag#911 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
Jun 12 01:27:54 rocky8 kernel: sd 0:0:0:0: [sda] tag#911 CDB: Read(10) 28 00 00 20 1b a0 00 00 08 00
Jun 12 01:27:54 rocky8 kernel: blk_update_request: I/O error, dev sda, sector 2104224 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Jun 12 01:27:54 rocky8 kernel: Read-error on swap-device (253:1:2976)
Also ZFS clearly segfaulted at the beginning. This could be related to the above or it could be an isolated case.
Jun 12 00:17:13 rocky8 kernel: ZFS: Loaded module v2.1.99-302_gffdf019cb (DEBUG mode), ZFS pool version 5000, ZFS filesystem version 5
Jun 12 00:17:17 rocky8 kernel: loop: module loaded
Jun 12 00:24:48 rocky8 kernel: lt-btree_test[48540]: segfault at 0 ip 00007f47eebb0aa3 sp 00007ffc6bf95990 error 4 in libzpool.so.5.0.0[7f47eeb2e000+38f000]
Jun 12 00:24:48 rocky8 kernel: Code: 57 41 56 49 89 f6 41 55 41 54 49 89 d4 53 48 89 fb 48 83 ec 58 48 8b 47 28 0f 29 45 90 48 c7 45 a0 00 00 00 00 48 85 c0 74 61 <48> 39 02 74 5c e8 33 f5 ff ff 4c 8d 65 90 4c 89 f6 48 89 df 4c 89
Please setup a similar CentOS 8 system under the following conditions:
* kernel 4.18.0-305.el8.x86_64 (do *not* update to 3.1 for now)
* Same system specs
* Same KVM settings
* Same ZFS module and version
* Using the same storage backend as your rocky VM
Run your tests on this system and provide the logs from journalctl -k. Also attach the XML configuration of both machines to this bug report.
Setting to NEEDINFO for now. |
|
(0000075)
Rich Ercolani
2021-06-12 09:44
|
This is Virtualbox, not KVM, and neither the host nor other guests have reported any similar issues.
I'll try setting up the Centos VM as described and get back to you, but this also happened on 4.18.0-240.22.1.el8.x86_64.
(The backend being used, as you might infer from some of the log messages, is virtio-scsi.)
(The ZFS test segfault is currently expected.) |
|
(0000076)
Rich Ercolani
2021-06-12 14:01
|
It took 2 runs on Centos 8 with 4.18.0-305.el8.x86_64, versus reproducing every time I tried on Rocky, but it did the same thing. |
|
(0000077)
Louis Abel
2021-06-12 21:51
|
I cannot reproduce your issue. I've ran the ZFS tests twice on a virtualbox system with virtio_scsi. I have attached a dmesg log from my own system. I will leave this ticket as NEEDINFO. |
|
(0000078)
Rich Ercolani
2021-06-13 15:03
|
1) The host VirtualBox system is Windows. Should that make a difference? No, but I'm mentioning it for completeness.
2) You used 0.8.6, while I was using git master. The number of commits between those two is...high. I can try and see if my case reproduces with 0.8.6 or 2.0.4 (which is in the zfs-testing repo in the same zfs repo file, where I assume you got your package), but I had something else I wanted to try first.
3) Most importantly, I too could not reproduce it...with one core given to the VM. With two cores, however, fire ensued. |
|
(0000079)
Rich Ercolani
2021-07-25 11:01
|
Incidental note, still happens on 4.18.0-305.10.2.el8_4.x86_64 |
|
(0000080)
Louis Abel
2022-04-22 06:17
|
reuploading attachments |
|
|
Notes |
(0000070)
Liam Hopkins
2022-04-08 18:36
|
thanks for quick reply. everything you said makes sense as a general EL derivative. i may have misfiled the bug, this is a request for a change to the google-specific rocky image being built by CIQ, which *does* have a custom kernel build. |
|
(0000071)
Louis Abel
2022-04-08 18:41
|
Acknowledged.
I would recommend that you reach out to CIQ directly as it is their kernel and thus their modifications. We do not have hands nor knowledge in what the company does or modifies away from the Rocky Linux base. |
|
(0000072)
Neil Hanlon
2022-04-08 18:43
|
Hi all - I can clear up a bit here.
This is going to be a contribution via the Cloud SIG - my apologies for the miscommunication as I asked Liam to file the ticket here.
Taking this one for myself. |
|
(0000073)
Mustafa Gezen
2022-04-08 20:06
|
Thanks Neil, and sorry for the confusion Louis. This is indeed about the SIG/Cloud kernel to improve Rocky on cloud platforms even if it diverges from upstream.
After talking to Neil, I'll be re-assigning this to myself and take the lead regarding SIG/Cloud kernel improvements until the initial release.
Also thanks for filing a bug Liam, this shouldn't require too much work hopefully. I'll keep in touch! |
|
(0000069)
Louis Abel
2022-04-20 08:28
|
Unfortunately we do not modify our kernels nor do we roll our own kernels.
From a Release Engineering standpoint, our kernels are to match what is upstream (RHEL) with only branding and secure boot changes. Red Hat has stated in their bugzilla (and other mediums) that reducing the page size back to 4K on RHEL 8 would introduce KABI incompatibility and breakage. This implies that backporting said change is not a supported option.
My recommendation would be to take the current kernel SRPM and test reducing the page size that way. If it does work, then it could be a potential addition/collaboration with SIG/kernel, but will unfortunately increase the amount of overhead in keeping all kernels in line and up to date as they come upstream. And as a result, can create various KABI incompatibilities with the rest of the distribution. But it will be up to you (and others) to test and confirm this to check if the pros outweigh the cons.
As an aside, this 64k page size started with various questions asking about Apple's M1 hardware (which is a limitation on their part). This (among other reasons such as drivers which do not work on non-standard page sizes) is why page sizes are being reduced for non-x86 architectures. |
|
|
Notes |
(0000106)
Lukas Magauer
2021-10-14 18:41
|
Install Source Error
Anaconda log from OpenQA with error |
|
(0000107)
Louis Abel
2021-10-14 19:14
|
I am unsure this is something that Release Engineering can fix. The .treeinfo is configured correctly as you can point to a direct mirror (such as dl.rl.o or anything that comes up in the list for your region) and it just works. The mirrorlist is behaving as it should as well, because dnf functions properly.
As this behavior *only* occurs while trying to use a mirror through anaconda, it will require Neil to assist looking into it, who is on vacation. |
|
(0000108)
Lukas Magauer
2021-10-14 22:52
|
Okay I'm totally at you, this shouldn't be that high on the priority list, we thought maybe something for 8.5.
It just looked like a bug in Anaconda to us.
But maybe it is also just a configuration thing about the MirrorManager. |
|
(0000109)
Louis Abel
2021-10-14 23:09
|
It could be either one. In 8.5 there is an anaconda version bump, so perhaps it has some fixes if it's not our mirror manager.
https://kojidev.rockylinux.org/koji/buildinfo?buildID=12442 |
|
(0000110)
Lukas Magauer
2021-10-15 14:49
|
I had access to another 8.5-beta system with anaconda 33.16.5.4, and I know that there it still doesn't work. Same behavior as with our 8.4 image.
As this is kind of a minor problem, I set it to that now. |
|
(0000111)
Neil Hanlon
2021-10-19 22:20
|
Oh goody. Anaconda debugging! |
|
|
Notes |
(0000082)
jonathan MERCIER
2021-08-12 15:19
|
I have not try the playbooks below, as I use directory architecture I can introduce some yaml syntax error but the general idea of playbook was rewritten, see below.
Too I try to switch to permissive mode but I have the same issue
```
---
- name: 'Install python3'
hosts: 'ipaserver'
become: true
gather_facts: false
tasks:
- name: 'Check if Python 3 is installed'
raw: 'python3 --version'
register: is_python_installed
ignore_errors: true
changed_when: is_python_installed.rc != 0
- name: 'Get OS ID like'
raw: 'source /etc/os-release && echo "${ID_LIKE}"'
register: id_like
when: is_python_installed.rc != 0
- name: 'Install python3 on rhel like os'
raw: 'yum install -y python3'
when: is_python_installed.rc != 0 and 'rhel' in id_like.stdout
- name: 'Install python3 on debian like os'
raw: 'apt update && apt install -y python3'
when: is_python_installed.rc != 0 and 'debian' in id_like.stdout
- names: 'freeipa_initialization'
hosts: 'ipaserver'
become: true
roles:
tasks:
- name: install firewalld
dnf:
name: firewalld
state: latest
- name: 'Remove ipa domain into host line with multiple domain name into /etc/hosts'
lineinfile:
path: '/etc/hosts'
regexp: '^(127\.0\.0\.1.+){{ inventory_hostname }}(\s*.+)$'
line: '\1\2'
backrefs: true
- name: 'Remove line where line describe single association of 127.0.0.1 and ipa domain into /etc/hosts'
lineinfile:
path: '/etc/hosts'
regexp: '^127\.0\.0\.1.+{{ inventory_hostname }}\s*$'
state: 'absent'
- name: 'Add association between external ip and ipa domain into etc/hosts'
lineinfile:
path: '/etc/hosts'
line: '{{ ipaserver_ip_addresses|first }} {{ inventory_hostname }}'
insertbefore: BOF
- name: Check hostname is valid
command: hostname -i
register: hostname_ip
failed_when: hostname_ip.stdout != ipaserver_ip_addresses|first
- name: 'Allow traffic in default zone for freeipa services'
ansible.posix.firewalld:
service: '{{ item }}'
permanent: true
state: 'enabled'
with_items:
- 'freeipa-ldap'
- 'freeipa-ldaps'
- 'ntp'
- 'dns'
- 'freeipa-4'
- name: 'Playbook to configure IPA servers'
hosts: 'ipaserver'
become: true
collections:
- 'freeipa.ansible_freeipa'
#vars_files:
# - 'group_vars/ipaserver'
# - 'group_vars/ipaserver_vault'
vars:
ipaserver_domain: 'infra.foo.com'
ipaserver_realm: 'INFRA.FOO.COM'
ipaserver_setup_dns: true
ipaserver_auto_forwarders: true
ipaserver_idstart: 2000
ipaserver_install_packages: true
ipaserver_ip_addresses:
- '{{ ansible_default_ipv4.address|default(ansible_all_ipv4_addresses[0]) }}'
ipaadmin_password: ADMPassword1
ipadm_password: DMPassword1
roles:
- role: 'ipaserver'
state: 'present'
``` |
|
(0000083)
jonathan MERCIER
2021-08-12 23:19
|
The issue can be reproduce more easily by calling ipa-server-install command. See below
maybe it is the same red hat bug: https://pagure.io/freeipa/issue/8907
```
# ipa-server-install --ds-password='changeme' --admin-password='changeme' --setup-dns --idstart=2000 --domain=infra.foo.com --realm=INFRA.FOO.COM --hostname=identity.foo.com --dirsrv-pin='changeme' --http-pin='changeme' --pkinit-pin='changeme' --mkhomedir --ntp-server=XX.YY.ZZ.II --auto-forwarders
...
INFO: Starting server\nDEBUG: Command: systemctl start pki-tomcatd@pki-tomcat.service\nINFO: FIPS mode: False\nINFO: Waiting for CA subsystem to start (1s)\nINFO: Waiting for CA subsystem to start (2s)\nINFO: Waiting for CA subsystem to start (3s)\nINFO: Waiting for CA subsystem to start (5s)\nINFO: Waiting for CA subsystem to start (6s)\nINFO: Waiting for CA subsystem to start (7s)\nINFO: Waiting for CA subsystem to start (8s)\nINFO: Waiting for CA subsystem to start (9s)\nINFO: Waiting for CA subsystem to start (10s)\nINFO: Waiting for CA subsystem to start (11s)\nINFO: Waiting for CA subsystem to start (12s)\nINFO: Waiting for CA subsystem to start (13s)\nINFO: Waiting for CA subsystem to start (14s)\nINFO: Waiting for CA subsystem to start (15s)\nINFO: Waiting for CA subsystem to start (16s)\nINFO: Waiting for CA subsystem to start (17s)\nINFO: Waiting for CA subsystem to start (19s)\nINFO: Waiting for CA subsystem to start (20s)\nINFO: Waiting for CA subsystem to start (21s)\nINFO: Waiting for CA subsystem to start (22s)\nINFO: Waiting for CA subsystem to start (23s)\nINFO: Waiting for CA subsystem to start (24s)\nINFO: Waiting for CA subsystem to start (25s)\nINFO: Waiting for CA subsystem to start (26s)\nINFO: Waiting for CA subsystem to start (27s)\nINFO: Waiting for CA subsystem to start (28s)\nINFO: Waiting for CA subsystem to start (29s)\nINFO: Waiting for CA subsystem to start (30s)\nINFO: Waiting for CA subsystem to start (31s)\nINFO: Waiting for CA subsystem to start (32s)\nINFO: Waiting for CA subsystem to start (33s)\nINFO: Waiting for CA subsystem to start (34s)\nINFO: Waiting for CA subsystem to start (35s)\nINFO: Waiting for CA subsystem to start (36s)\nINFO: Waiting for CA subsystem to start (38s)\nINFO: Waiting for CA subsystem to start (39s)\nINFO: Waiting for CA subsystem to start (40s)\nINFO: Waiting for CA subsystem to start (41s)\nINFO: Waiting for CA subsystem to start (42s)\nINFO: Waiting for CA subsystem to start (43s)\nINFO: Waiting for CA subsystem to start (44s)\nINFO: Waiting for CA subsystem to start (45s)\nINFO: Waiting for CA subsystem to start (46s)\nINFO: Waiting for CA subsystem to start (47s)\nINFO: Waiting for CA subsystem to start (48s)\nINFO: Waiting for CA subsystem to start (49s)\nINFO: Waiting for CA subsystem to start (50s)\nINFO: Waiting for CA subsystem to start (51s)\nINFO: Waiting for CA subsystem to start (52s)\nINFO: Waiting for CA subsystem to start (53s)\nINFO: Waiting for CA subsystem to start (54s)\nINFO: Waiting for CA subsystem to start (55s)\nINFO: Waiting for CA subsystem to start (56s)\nINFO: Waiting for CA subsystem to start (57s)\nINFO: Waiting for CA subsystem to start (58s)\nINFO: Waiting for CA subsystem to start (59s)\nERROR: Exception: CA subsystem did not start after 60s\n File "/usr/lib/python3.6/site-packages/pki/server/pkispawn.py", line 575, in main\n scriptlet.spawn(deployer)\n File "/usr/lib/python3.6/site-packages/pki/server/deployment/scriptlets/configuration.py", line 965, in spawn\n request_timeout,\n File "/usr/lib/python3.6/site-packages/pki/server/deployment/pkihelper.py", line 891, in wait_for_startup\n (subsystem.type, startup_timeout)) from exc\n\n')
See the installation logs and the following files/directories for more information:
/var/log/pki/pki-tomcat
[error] RuntimeError: CA configuration failed.
CA configuration failed.
The ipa-server-install command failed. See /var/log/ipaserver-install.log for more information
``` |
|
(0000084)
Louis Abel
2021-08-13 01:10
|
Hello.
The pagure link in #2 refers to when CentOS did their build against a newer NSS (stream) instead of the current NSS (in 8.4). We did not have this issue and thus did not need to rebuild.
Please install the rpaste package and provide the output of rpaste --sysinfo. Optionally, you can provide us this information manually:
CPU
RAM
Disk + Partition and volume layout
cat /etc/os-release |
|
(0000085)
jonathan MERCIER
2021-08-13 07:13
|
Thanks Louis Abel for your help.
Here the requested information: https://rpa.st/DV5A |
|
(0000086)
jonathan MERCIER
2021-08-13 07:30
|
Some extra contextual information that could help (or not):
While the pki-tomcatd service is running we can see somme errors
```
# LANG=C journalctl -xe -u pki-tomcatd@pki-tomcat
-- Logs begin at Thu 2021-08-12 23:50:54 CEST, end at Fri 2021-08-13 09:16:20 CEST. --
Aug 13 00:42:59 identity.foo.com systemd[1]: Starting PKI Tomcat Server pki-tomcat...
-- Subject: Unit pki-tomcatd@pki-tomcat.service has begun start-up
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit pki-tomcatd@pki-tomcat.service has begun starting up.
Aug 13 00:43:02 identity.foo.com java[69998]: usr/lib/api/apiutil.c Could not open /run/lock/opencryptoki/LCK..APIlock
Aug 13 00:43:04 identity.foo.com systemd[1]: Started PKI Tomcat Server pki-tomcat.
-- Subject: Unit pki-tomcatd@pki-tomcat.service has finished start-up
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit pki-tomcatd@pki-tomcat.service has finished starting up.
--
-- The start-up result is done.
Aug 13 00:43:04 identity.foo.com server[70104]: Java virtual machine used: /usr/lib/jvm/java-1.8.0-openjdk/bin/java
Aug 13 00:43:04 identity.foo.com server[70104]: classpath used: /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/ant.jar:/usr/share/java/ant-launcher.j>
Aug 13 00:43:04 identity.foo.com server[70104]: main class used: org.apache.catalina.startup.Bootstrap
Aug 13 00:43:04 identity.foo.com server[70104]: flags used: -Dcom.redhat.fips=false
Aug 13 00:43:04 identity.foo.com server[70104]: options used: -Dcatalina.base=/var/lib/pki/pki-tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/lib/pki/pki->
Aug 13 00:43:04 identity.foo.com server[70104]: arguments used: start
Aug 13 00:43:05 identity.foo.com java[70104]: usr/lib/api/apiutil.c Could not open /run/lock/opencryptoki/LCK..APIlock
Aug 13 00:43:06 identity.foo.com server[70104]: WARNING: Some of the specified [protocols] are not supported by the SSL engine and have been skipped: [[TLSv1, TLSv1.1]]
# systemctl status pki-tomcatd@pki-tomcat.service
● pki-tomcatd@pki-tomcat.service - PKI Tomcat Server pki-tomcat
Loaded: loaded (/usr/lib/systemd/system/pki-tomcatd@.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2021-08-13 00:43:04 CEST; 8h ago
Process: 70013 ExecStartPre=/usr/bin/pkidaemon start pki-tomcat (code=exited, status=0/SUCCESS)
Process: 69980 ExecStartPre=/usr/sbin/pki-server migrate pki-tomcat (code=exited, status=0/SUCCESS)
Process: 69977 ExecStartPre=/usr/sbin/pki-server upgrade pki-tomcat (code=exited, status=0/SUCCESS)
Main PID: 70104 (java)
Tasks: 115 (limit: 23448)
Memory: 465.5M
CGroup: /system.slice/system-pki\x2dtomcatd.slice/pki-tomcatd@pki-tomcat.service
└─70104 /usr/lib/jvm/java-1.8.0-openjdk/bin/java -Dcom.redhat.fips=false -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/ant.jar:/usr/shar>
```
Indeed /run/lock/opencryptoki/LCK..APIlock is not present
```
# ls /run/lock/opencryptoki/LCK..APIlock
ls: cannot access '/run/lock/opencryptoki/LCK..APIlock': No such file or directory
# ls /run/lock/opencryptoki
icsf swtok tpm
```
This file is usualy generated by pkcsslotd service ... and this service is dead
```
# systemctl status pkcsslotd
* pkcsslotd.service - Daemon which manages cryptographic hardware tokens for the openCryptoki package
Loaded: loaded (/usr/lib/systemd/system/pkcsslotd.service; disabled; vendor preset: disabled)
Active: inactive (dead)
# systemctl start pkcsslotd
# systemctl status pkcsslotd
* pkcsslotd.service - Daemon which manages cryptographic hardware tokens for the openCryptoki package
Loaded: loaded (/usr/lib/systemd/system/pkcsslotd.service; disabled; vendor preset: disabled)
Active: active (running) since Fri 2021-08-13 09:25:35 CEST; 1s ago
Process: 71344 ExecStart=/usr/sbin/pkcsslotd (code=exited, status=0/SUCCESS)
Main PID: 71345 (pkcsslotd)
Tasks: 1 (limit: 23448)
Memory: 5.6M
CGroup: /system.slice/pkcsslotd.service
`-71345 /usr/sbin/pkcsslotd
Aug 13 09:25:34 identity.microbiome.studio systemd[1]: Starting Daemon which manages cryptographic hardware tokens for the openCryptoki package...
Aug 13 09:25:35 identity.microbiome.studio systemd[1]: Started Daemon which manages cryptographic hardware tokens for the openCryptoki package.
# ls /run/lock/opencryptoki/LCK..APIlock
/run/lock/opencryptoki/LCK..APIlock
``` |
|
(0000087)
jonathan MERCIER
2021-08-13 10:38
|
After read this freeipa user lists: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org/thread/XFYVC6MUAKYLRIR6H6WM6SD4USLMIG2E/
I removed from /etc/crypto-policies/back-ends/nss.config these two lines:
name=p11-kit-proxy
library=p11-kit-proxy.so
And I downgraded 389-ds-base:
dnf downgrade -y 389-ds-base
With this I get rid of error:
usr/lib/api/apiutil.c Could not open /run/lock/opencryptoki/LCK..APIlock
Ass seen here:
```
# systemctl status pki-tomcatd@pki-tomcat
● pki-tomcatd@pki-tomcat.service - PKI Tomcat Server pki-tomcat
Loaded: loaded (/usr/lib/systemd/system/pki-tomcatd@.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2021-08-13 12:32:13 CEST; 1min 31s ago
Process: 11182 ExecStartPre=/usr/bin/pkidaemon start pki-tomcat (code=exited, status=0/SUCCESS)
Process: 11149 ExecStartPre=/usr/sbin/pki-server migrate pki-tomcat (code=exited, status=0/SUCCESS)
Process: 11146 ExecStartPre=/usr/sbin/pki-server upgrade pki-tomcat (code=exited, status=0/SUCCESS)
Main PID: 11273 (java)
Tasks: 115 (limit: 23441)
Memory: 475.3M
CGroup: /system.slice/system-pki\x2dtomcatd.slice/pki-tomcatd@pki-tomcat.service
└─11273 /usr/lib/jvm/java-1.8.0-openjdk/bin/java -Dcom.redhat.fips=false -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/ant.jar:/usr/shar>
août 13 12:32:08 identity.microbiome.studio systemd[1]: Starting PKI Tomcat Server pki-tomcat...
août 13 12:32:13 identity.microbiome.studio systemd[1]: Started PKI Tomcat Server pki-tomcat.
août 13 12:32:13 identity.microbiome.studio server[11273]: Java virtual machine used: /usr/lib/jvm/java-1.8.0-openjdk/bin/java
août 13 12:32:13 identity.microbiome.studio server[1173]: classpath used: /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/ant.jar:/usr/share/java/ant-launcher.>
août 13 12:32:13 identity.microbiome.studio server[11273]: main class used: org.apache.catalina.startup.Bootstrap
août 13 12:32:13 identity.microbiome.studio server[11273]: flags used: -Dcom.redhat.fips=false
août 13 12:32:13 identity.microbiome.studio server[11273]: options used: -Dcatalina.base=/var/lib/pki/pki-tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/lib/pki/pki>
août 13 12:32:13 identity.microbiome.studio server[11273]: arguments used: start
août 13 12:32:16 identity.microbiome.studio server[11273]: WARNING: Some of the specified [protocols] are not supported by the SSL engine and have been skipped: [[TLSv1, TLSv1.1]]
```
but the process still fail
```
INFO: Starting server\nDEBUG: Command: systemctl start pki-tomcatd@pki-tomcat.service\nINFO: FIPS mode: False\nINFO: Waiting for CA subsystem to start (1s)\nINFO: Waiting for CA subsystem to start (2s)\nINFO: Waiting for CA subsystem to start (3s)\nINFO: Waiting for CA subsystem to start (4s)\nINFO: Waiting for CA subsystem to start (5s)\nINFO: Waiting for CA subsystem to start (6s)\nINFO: Waiting for CA subsystem to start (7s)\nINFO: Waiting for CA subsystem to start (8s)\nINFO: Waiting for CA subsystem to start (9s)\nINFO: Waiting for CA subsystem to start (10s)\nINFO: Waiting for CA subsystem to start (11s)\nINFO: Waiting for CA subsystem to start (12s)\nINFO: Waiting for CA subsystem to start (13s)\nINFO: Waiting for CA subsystem to start (14s)\nINFO: Waiting for CA subsystem to start (15s)\nINFO: Waiting for CA subsystem to start (16s)\nINFO: Waiting for CA subsystem to start (18s)\nINFO: Waiting for CA subsystem to start (19s)\nINFO: Waiting for CA subsystem to start (20s)\nINFO: Waiting for CA subsystem to start (21s)\nINFO: Waiting for CA subsystem to start (22s)\nINFO: Waiting for CA subsystem to start (23s)\nINFO: Waiting for CA subsystem to start (24s)\nINFO: Waiting for CA subsystem to start (25s)\nINFO: Waiting for CA subsystem to start (26s)\nINFO: Waiting for CA subsystem to start (27s)\nINFO: Waiting for CA subsystem to start (28s)\nINFO: Waiting for CA subsystem to start (29s)\nINFO: Waiting for CA subsystem to start (30s)\nINFO: Waiting for CA subsystem to start (31s)\nINFO: Waiting for CA subsystem to start (32s)\nINFO: Waiting for CA subsystem to start (33s)\nINFO: Waiting for CA subsystem to start (34s)\nINFO: Waiting for CA subsystem to start (35s)\nINFO: Waiting for CA subsystem to start (36s)\nINFO: Waiting for CA subsystem to start (37s)\nINFO: Waiting for CA subsystem to start (38s)\nINFO: Waiting for CA subsystem to start (39s)\nINFO: Waiting for CA subsystem to start (40s)\nINFO: Waiting for CA subsystem to start (41s)\nINFO: Waiting for CA subsystem to start (42s)\nINFO: Waiting for CA subsystem to start (43s)\nINFO: Waiting for CA subsystem to start (44s)\nINFO: Waiting for CA subsystem to start (45s)\nINFO: Waiting for CA subsystem to start (46s)\nINFO: Waiting for CA subsystem to start (47s)\nINFO: Waiting for CA subsystem to start (48s)\nINFO: Waiting for CA subsystem to start (49s)\nINFO: Waiting for CA subsystem to start (50s)\nINFO: Waiting for CA subsystem to start (51s)\nINFO: Waiting for CA subsystem to start (52s)\nINFO: Waiting for CA subsystem to start (53s)\nINFO: Waiting for CA subsystem to start (54s)\nINFO: Waiting for CA subsystem to start (55s)\nINFO: Waiting for CA subsystem to start (56s)\nINFO: Waiting for CA subsystem to start (57s)\nINFO: Waiting for CA subsystem to start (58s)\nINFO: Waiting for CA subsystem to start (60s)\nERROR: Exception: CA subsystem did not start after 60s\n File "/usr/lib/python3.6/site-packages/pki/server/pkispawn.py", line 575, in main\n scriptlet.spawn(deployer)\n File "/usr/lib/python3.6/site-packages/pki/server/deployment/scriptlets/configuration.py", line 965, in spawn\n request_timeout,\n File "/usr/lib/python3.6/site-packages/pki/server/deployment/pkihelper.py", line 891, in wait_for_startup\n (subsystem.type, startup_timeout)) from exc\n\n')
See the installation logs and the following files/directories for more information:
/var/log/pki/pki-tomcat
[error] RuntimeError: CA configuration failed.
``` |
|