 |
We also have this issue, or one very much like it. We have a busy fileserver that upgraded from Rocky 9.4 to 9.5 this morning at 0540. It returned, but at 0610 stopped responding. The console was unresponsive so we hard rebooted via vm controls. Within 30 minutes of that, had stopped again. We hard booted again, and this time chose the previous kernel. Since then (5h+) the vm has been 100% stable as it has been before 9.5 It serves files through both NFSv4 and SMB, using SSSD and AD for authentication. I also get the impression that it's related to NFS traffic. Our logs follow Dec 3 05:53:41 redacted_hostname: [ 769.252982] ------------[ cut here ]------------ Dec 3 05:53:41 redacted_hostname: [ 769.252987] WARNING: CPU: 0 PID: 1115 at mm/slab_common.c:957 free_large_kmalloc+0x5a/0x80 Dec 3 05:53:41 redacted_hostname: [ 769.253003] Modules linked in: binfmt_misc rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache netfs rpcrdma rdma_cm iw_cm ib_cm ib_core rfkill nft_reject_ipv4 nf_reject_ipv4 nft_reject nft_counter nft_ct nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 vsock_loopback vmw_vsock_virtio_transport_common nf_tables vmw_vsock_vmci_transport vsock nfnetlink vmwgfx intel_rapl_msr vmw_balloon intel_rapl_common drm_ttm_helper ttm vmw_vmci pcspkr drm_kms_helper i2c_piix4 joydev nfsd nfs_acl lockd auth_rpcgss grace sunrpc drm xfs libcrc32c crct10dif_pclmul sd_mod crc32_pclmul ata_generic t10_pi crc32c_intel sg ghash_clmulni_intel ata_piix libata vmxnet3 vmw_pvscsi serio_raw dm_mirror dm_region_hash dm_log dm_mod fuse Dec 3 05:53:41 redacted_hostname: [ 769.253054] CPU: 0 PID: 1115 Comm: nfsd Not tainted 5.14.0-503.15.1.el9_5.x86_64 #1 Dec 3 05:53:41 redacted_hostname: [ 769.253056] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020 Dec 3 05:53:41 redacted_hostname: [ 769.253057] RIP: 0010:free_large_kmalloc+0x5a/0x80 Dec 3 05:53:41 redacted_hostname: [ 769.253060] Code: da 9c 5b fa be 06 00 00 00 48 89 ef e8 af 25 0a 00 80 e7 02 74 01 fb 48 83 c4 08 44 89 e6 48 89 ef 5b 5d 41 5c e9 d6 28 04 00 <0f> 0b 45 31 e4 80 3d d3 0d fc 01 00 ba 00 f0 ff ff 0f 84 8b 9a 90 Dec 3 05:53:41 redacted_hostname: [ 769.253062] RSP: 0018:ffffafef00d57b28 EFLAGS: 00010246 Dec 3 05:53:41 redacted_hostname: [ 769.253064] RAX: 0017ffffc0000000 RBX: ffffffff8cd1e2e0 RCX: ffff9fce52f7ac68 Dec 3 05:53:41 redacted_hostname: [ 769.253065] RDX: dead000000000100 RSI: ffffffffc096b47c RDI: ffffd9fd06c25ac0 Dec 3 05:53:41 redacted_hostname: [ 769.253065] RBP: ffffd9fd06c25ac0 R08: ffffffff8d4075e0 R09: ffff9fcf35e344b0 Dec 3 05:53:41 redacted_hostname: [ 769.253066] R10: 000000b30f218140 R11: 00000000004c82ea R12: ffffffffc096b47c Dec 3 05:53:41 redacted_hostname: [ 769.253067] R13: 0000000000000000 R14: ffff9fce0a0c4a00 R15: ffff9fce01148000 Dec 3 05:53:41 redacted_hostname: [ 769.253068] FS: 0000000000000000(0000) GS:ffff9fcf35e00000(0000) knlGS:0000000000000000 Dec 3 05:53:41 redacted_hostname: [ 769.253070] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Dec 3 05:53:41 redacted_hostname: [ 769.253071] CR2: 00007efd2f51a654 CR3: 000000010902a002 CR4: 00000000007706f0 Dec 3 05:53:41 redacted_hostname: [ 769.253085] PKRU: 55555554 Dec 3 05:53:41 redacted_hostname: [ 769.253086] Call Trace: Dec 3 05:53:41 redacted_hostname: [ 769.253087] <TASK> Dec 3 05:53:41 redacted_hostname: [ 769.253088] ? srso_alias_return_thunk+0x5/0xfbef5 Dec 3 05:53:41 redacted_hostname: [ 769.253094] ? show_trace_log_lvl+0x26e/0x2df Dec 3 05:53:41 redacted_hostname: [ 769.253101] ? show_trace_log_lvl+0x26e/0x2df Dec 3 05:53:41 redacted_hostname: [ 769.253106] ? security_release_secctx+0x28/0x40 Dec 3 05:53:41 redacted_hostname: [ 769.253110] ? free_large_kmalloc+0x5a/0x80 Dec 3 05:53:41 redacted_hostname: [ 769.253113] ? __warn+0x7e/0xd0 Dec 3 05:53:41 redacted_hostname: [ 769.253116] ? free_large_kmalloc+0x5a/0x80 Dec 3 05:53:41 redacted_hostname: [ 769.253119] ? report_bug+0x100/0x140 Dec 3 05:53:41 redacted_hostname: [ 769.253124] ? handle_bug+0x3c/0x70 Dec 3 05:53:41 redacted_hostname: [ 769.253127] ? exc_invalid_op+0x14/0x70 Dec 3 05:53:41 redacted_hostname: [ 769.253130] ? asm_exc_invalid_op+0x16/0x20 Dec 3 05:53:41 redacted_hostname: [ 769.253134] ? _fh_update.part.0.isra.0+0x4c/0x90 [nfsd] Dec 3 05:53:41 redacted_hostname: [ 769.253162] ? _fh_update.part.0.isra.0+0x4c/0x90 [nfsd] Dec 3 05:53:41 redacted_hostname: [ 769.253184] ? free_large_kmalloc+0x5a/0x80 Dec 3 05:53:41 redacted_hostname: [ 769.253188] ? _fh_update.part.0.isra.0+0x4c/0x90 [nfsd] Dec 3 05:53:41 redacted_hostname: [ 769.253206] security_release_secctx+0x28/0x40 Dec 3 05:53:41 redacted_hostname: [ 769.253209] nfsd4_encode_fattr4+0x2cc/0x4f0 [nfsd] Dec 3 05:53:41 redacted_hostname: [ 769.253237] ? srso_alias_return_thunk+0x5/0xfbef5 Dec 3 05:53:41 redacted_hostname: [ 769.253239] ? __kmem_cache_alloc_node+0x18f/0x2e0 Dec 3 05:53:41 redacted_hostname: [ 769.253242] ? security_prepare_creds+0x71/0xa0 Dec 3 05:53:41 redacted_hostname: [ 769.253245] ? security_prepare_creds+0x71/0xa0 Dec 3 05:53:41 redacted_hostname: [ 769.253242] ? security_prepare_creds+0x71/0xa0 Dec 3 05:53:41 redacted_hostname: [ 769.253245] ? security_prepare_creds+0x71/0xa0 Dec 3 05:53:41 redacted_hostname: [ 769.253246] ? srso_alias_return_thunk+0x5/0xfbef5 Dec 3 05:53:41 redacted_hostname: [ 769.253248] ? __kmalloc+0x4b/0x140 Dec 3 05:53:41 redacted_hostname: [ 769.253250] ? srso_alias_return_thunk+0x5/0xfbef5 Dec 3 05:53:41 redacted_hostname: [ 769.253251] ? srso_alias_return_thunk+0x5/0xfbef5 Dec 3 05:53:41 redacted_hostname: [ 769.253253] ? security_prepare_creds+0x47/0xa0 Dec 3 05:53:41 redacted_hostname: [ 769.253255] ? srso_alias_return_thunk+0x5/0xfbef5 Dec 3 05:53:41 redacted_hostname: [ 769.253256] ? prepare_creds+0x180/0x270 Dec 3 05:53:41 redacted_hostname: [ 769.253259] ? srso_alias_return_thunk+0x5/0xfbef5 Dec 3 05:53:41 redacted_hostname: [ 769.253261] ? nfsd_setuser+0x110/0x270 [nfsd] Dec 3 05:53:41 redacted_hostname: [ 769.253286] ? srso_alias_return_thunk+0x5/0xfbef5 Dec 3 05:53:41 redacted_hostname: [ 769.253288] ? nfsd_setuser_and_check_port+0x4a/0xc0 [nfsd] Dec 3 05:53:41 redacted_hostname: [ 769.253306] ? _fh_update.part.0.isra.0+0x4c/0x90 [nfsd] Dec 3 05:53:41 redacted_hostname: [ 769.253323] nfsd4_encode_getattr+0x2b/0x40 [nfsd] Dec 3 05:53:41 redacted_hostname: [ 769.253341] nfsd4_encode_operation+0xa6/0x2b0 [nfsd] Dec 3 05:53:41 redacted_hostname: [ 769.253361] nfsd4_proc_compound+0x1d0/0x700 [nfsd] Dec 3 05:53:41 redacted_hostname: [ 769.253384] nfsd_dispatch+0xe9/0x220 [nfsd] Dec 3 05:53:41 redacted_hostname: [ 769.253404] svc_process_common+0x2e7/0x650 [sunrpc] Dec 3 05:53:41 redacted_hostname: [ 769.253435] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd] Dec 3 05:53:41 redacted_hostname: [ 769.253463] svc_process+0x12d/0x170 [sunrpc] Dec 3 05:53:41 redacted_hostname: [ 769.253491] svc_handle_xprt+0x448/0x580 [sunrpc] Dec 3 05:53:41 redacted_hostname: [ 769.253523] svc_recv+0x17a/0x2c0 [sunrpc] Dec 3 05:53:41 redacted_hostname: [ 769.253552] ? __pfx_nfsd+0x10/0x10 [nfsd] Dec 3 05:53:41 redacted_hostname: [ 769.253577] nfsd+0x84/0xb0 [nfsd] Dec 3 05:53:41 redacted_hostname: [ 769.253600] kthread+0xe0/0x100 Dec 3 05:53:41 redacted_hostname: [ 769.253604] ? __pfx_kthread+0x10/0x10 Dec 3 05:53:41 redacted_hostname: [ 769.253608] ret_from_fork+0x2c/0x50 Dec 3 05:53:41 redacted_hostname: [ 769.253614] </TASK> Dec 3 05:53:41 redacted_hostname: [ 769.253614] ---[ end trace 0000000000000000 ]--- Dec 3 05:53:41 redacted_hostname: [ 769.253616] object pointer: 0x000000008b742c83 |
 |
https://nvd.nist.gov/vuln/detail/CVE-2024-46697 feels related |
|
|
Would either of you have access or ability to generate a kdump? I'd like to poke around and see if my assumptions here are correct.. but I believe this crash is being triggered by the lack of the fix for CVE-2024-46697 (https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=f58bab6fd4063913bd8321e99874b8239e9ba726) in the 9.5 kernel. i.e., the specified condition occurs and args.context is set with random junk which, when the kernel attempts to free, sometimes causes a crash -- the effect thereof would be pronounced further on busy servers, probably. |
|
|
In /var/crash i have a lot of kexec-dmesg.log, vmcore, vmcore-dmesg.txt Is that what you would need? |
|
|
https://issues.redhat.com/browse/RHEL-69877 Opened this for now |
- Anonymous
