 |
We also have this issue, or one very much like it. We have a busy fileserver that upgraded from Rocky 9.4 to 9.5 this morning at 0540. It returned, but at 0610 stopped responding. The console was unresponsive so we hard rebooted via vm controls. Within 30 minutes of that, had stopped again. We hard booted again, and this time chose the previous kernel. Since then (5h+) the vm has been 100% stable as it has been before 9.5 It serves files through both NFSv4 and SMB, using SSSD and AD for authentication. I also get the impression that it's related to NFS traffic. Our logs follow Dec 3 05:53:41 redacted_hostname: [ 769.252982] ------------[ cut here ]------------ Dec 3 05:53:41 redacted_hostname: [ 769.252987] WARNING: CPU: 0 PID: 1115 at mm/slab_common.c:957 free_large_kmalloc+0x5a/0x80 Dec 3 05:53:41 redacted_hostname: [ 769.253003] Modules linked in: binfmt_misc rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache netfs rpcrdma rdma_cm iw_cm ib_cm ib_core rfkill nft_reject_ipv4 nf_reject_ipv4 nft_reject nft_counter nft_ct nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 vsock_loopback vmw_vsock_virtio_transport_common nf_tables vmw_vsock_vmci_transport vsock nfnetlink vmwgfx intel_rapl_msr vmw_balloon intel_rapl_common drm_ttm_helper ttm vmw_vmci pcspkr drm_kms_helper i2c_piix4 joydev nfsd nfs_acl lockd auth_rpcgss grace sunrpc drm xfs libcrc32c crct10dif_pclmul sd_mod crc32_pclmul ata_generic t10_pi crc32c_intel sg ghash_clmulni_intel ata_piix libata vmxnet3 vmw_pvscsi serio_raw dm_mirror dm_region_hash dm_log dm_mod fuse Dec 3 05:53:41 redacted_hostname: [ 769.253054] CPU: 0 PID: 1115 Comm: nfsd Not tainted 5.14.0-503.15.1.el9_5.x86_64 #1 Dec 3 05:53:41 redacted_hostname: [ 769.253056] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020 Dec 3 05:53:41 redacted_hostname: [ 769.253057] RIP: 0010:free_large_kmalloc+0x5a/0x80 Dec 3 05:53:41 redacted_hostname: [ 769.253060] Code: da 9c 5b fa be 06 00 00 00 48 89 ef e8 af 25 0a 00 80 e7 02 74 01 fb 48 83 c4 08 44 89 e6 48 89 ef 5b 5d 41 5c e9 d6 28 04 00 <0f> 0b 45 31 e4 80 3d d3 0d fc 01 00 ba 00 f0 ff ff 0f 84 8b 9a 90 Dec 3 05:53:41 redacted_hostname: [ 769.253062] RSP: 0018:ffffafef00d57b28 EFLAGS: 00010246 Dec 3 05:53:41 redacted_hostname: [ 769.253064] RAX: 0017ffffc0000000 RBX: ffffffff8cd1e2e0 RCX: ffff9fce52f7ac68 Dec 3 05:53:41 redacted_hostname: [ 769.253065] RDX: dead000000000100 RSI: ffffffffc096b47c RDI: ffffd9fd06c25ac0 Dec 3 05:53:41 redacted_hostname: [ 769.253065] RBP: ffffd9fd06c25ac0 R08: ffffffff8d4075e0 R09: ffff9fcf35e344b0 Dec 3 05:53:41 redacted_hostname: [ 769.253066] R10: 000000b30f218140 R11: 00000000004c82ea R12: ffffffffc096b47c Dec 3 05:53:41 redacted_hostname: [ 769.253067] R13: 0000000000000000 R14: ffff9fce0a0c4a00 R15: ffff9fce01148000 Dec 3 05:53:41 redacted_hostname: [ 769.253068] FS: 0000000000000000(0000) GS:ffff9fcf35e00000(0000) knlGS:0000000000000000 Dec 3 05:53:41 redacted_hostname: [ 769.253070] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Dec 3 05:53:41 redacted_hostname: [ 769.253071] CR2: 00007efd2f51a654 CR3: 000000010902a002 CR4: 00000000007706f0 Dec 3 05:53:41 redacted_hostname: [ 769.253085] PKRU: 55555554 Dec 3 05:53:41 redacted_hostname: [ 769.253086] Call Trace: Dec 3 05:53:41 redacted_hostname: [ 769.253087] <TASK> Dec 3 05:53:41 redacted_hostname: [ 769.253088] ? srso_alias_return_thunk+0x5/0xfbef5 Dec 3 05:53:41 redacted_hostname: [ 769.253094] ? show_trace_log_lvl+0x26e/0x2df Dec 3 05:53:41 redacted_hostname: [ 769.253101] ? show_trace_log_lvl+0x26e/0x2df Dec 3 05:53:41 redacted_hostname: [ 769.253106] ? security_release_secctx+0x28/0x40 Dec 3 05:53:41 redacted_hostname: [ 769.253110] ? free_large_kmalloc+0x5a/0x80 Dec 3 05:53:41 redacted_hostname: [ 769.253113] ? __warn+0x7e/0xd0 Dec 3 05:53:41 redacted_hostname: [ 769.253116] ? free_large_kmalloc+0x5a/0x80 Dec 3 05:53:41 redacted_hostname: [ 769.253119] ? report_bug+0x100/0x140 Dec 3 05:53:41 redacted_hostname: [ 769.253124] ? handle_bug+0x3c/0x70 Dec 3 05:53:41 redacted_hostname: [ 769.253127] ? exc_invalid_op+0x14/0x70 Dec 3 05:53:41 redacted_hostname: [ 769.253130] ? asm_exc_invalid_op+0x16/0x20 Dec 3 05:53:41 redacted_hostname: [ 769.253134] ? _fh_update.part.0.isra.0+0x4c/0x90 [nfsd] Dec 3 05:53:41 redacted_hostname: [ 769.253162] ? _fh_update.part.0.isra.0+0x4c/0x90 [nfsd] Dec 3 05:53:41 redacted_hostname: [ 769.253184] ? free_large_kmalloc+0x5a/0x80 Dec 3 05:53:41 redacted_hostname: [ 769.253188] ? _fh_update.part.0.isra.0+0x4c/0x90 [nfsd] Dec 3 05:53:41 redacted_hostname: [ 769.253206] security_release_secctx+0x28/0x40 Dec 3 05:53:41 redacted_hostname: [ 769.253209] nfsd4_encode_fattr4+0x2cc/0x4f0 [nfsd] Dec 3 05:53:41 redacted_hostname: [ 769.253237] ? srso_alias_return_thunk+0x5/0xfbef5 Dec 3 05:53:41 redacted_hostname: [ 769.253239] ? __kmem_cache_alloc_node+0x18f/0x2e0 Dec 3 05:53:41 redacted_hostname: [ 769.253242] ? security_prepare_creds+0x71/0xa0 Dec 3 05:53:41 redacted_hostname: [ 769.253245] ? security_prepare_creds+0x71/0xa0 Dec 3 05:53:41 redacted_hostname: [ 769.253242] ? security_prepare_creds+0x71/0xa0 Dec 3 05:53:41 redacted_hostname: [ 769.253245] ? security_prepare_creds+0x71/0xa0 Dec 3 05:53:41 redacted_hostname: [ 769.253246] ? srso_alias_return_thunk+0x5/0xfbef5 Dec 3 05:53:41 redacted_hostname: [ 769.253248] ? __kmalloc+0x4b/0x140 Dec 3 05:53:41 redacted_hostname: [ 769.253250] ? srso_alias_return_thunk+0x5/0xfbef5 Dec 3 05:53:41 redacted_hostname: [ 769.253251] ? srso_alias_return_thunk+0x5/0xfbef5 Dec 3 05:53:41 redacted_hostname: [ 769.253253] ? security_prepare_creds+0x47/0xa0 Dec 3 05:53:41 redacted_hostname: [ 769.253255] ? srso_alias_return_thunk+0x5/0xfbef5 Dec 3 05:53:41 redacted_hostname: [ 769.253256] ? prepare_creds+0x180/0x270 Dec 3 05:53:41 redacted_hostname: [ 769.253259] ? srso_alias_return_thunk+0x5/0xfbef5 Dec 3 05:53:41 redacted_hostname: [ 769.253261] ? nfsd_setuser+0x110/0x270 [nfsd] Dec 3 05:53:41 redacted_hostname: [ 769.253286] ? srso_alias_return_thunk+0x5/0xfbef5 Dec 3 05:53:41 redacted_hostname: [ 769.253288] ? nfsd_setuser_and_check_port+0x4a/0xc0 [nfsd] Dec 3 05:53:41 redacted_hostname: [ 769.253306] ? _fh_update.part.0.isra.0+0x4c/0x90 [nfsd] Dec 3 05:53:41 redacted_hostname: [ 769.253323] nfsd4_encode_getattr+0x2b/0x40 [nfsd] Dec 3 05:53:41 redacted_hostname: [ 769.253341] nfsd4_encode_operation+0xa6/0x2b0 [nfsd] Dec 3 05:53:41 redacted_hostname: [ 769.253361] nfsd4_proc_compound+0x1d0/0x700 [nfsd] Dec 3 05:53:41 redacted_hostname: [ 769.253384] nfsd_dispatch+0xe9/0x220 [nfsd] Dec 3 05:53:41 redacted_hostname: [ 769.253404] svc_process_common+0x2e7/0x650 [sunrpc] Dec 3 05:53:41 redacted_hostname: [ 769.253435] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd] Dec 3 05:53:41 redacted_hostname: [ 769.253463] svc_process+0x12d/0x170 [sunrpc] Dec 3 05:53:41 redacted_hostname: [ 769.253491] svc_handle_xprt+0x448/0x580 [sunrpc] Dec 3 05:53:41 redacted_hostname: [ 769.253523] svc_recv+0x17a/0x2c0 [sunrpc] Dec 3 05:53:41 redacted_hostname: [ 769.253552] ? __pfx_nfsd+0x10/0x10 [nfsd] Dec 3 05:53:41 redacted_hostname: [ 769.253577] nfsd+0x84/0xb0 [nfsd] Dec 3 05:53:41 redacted_hostname: [ 769.253600] kthread+0xe0/0x100 Dec 3 05:53:41 redacted_hostname: [ 769.253604] ? __pfx_kthread+0x10/0x10 Dec 3 05:53:41 redacted_hostname: [ 769.253608] ret_from_fork+0x2c/0x50 Dec 3 05:53:41 redacted_hostname: [ 769.253614] </TASK> Dec 3 05:53:41 redacted_hostname: [ 769.253614] ---[ end trace 0000000000000000 ]--- Dec 3 05:53:41 redacted_hostname: [ 769.253616] object pointer: 0x000000008b742c83 |
 |
https://nvd.nist.gov/vuln/detail/CVE-2024-46697 feels related |
|
|
Would either of you have access or ability to generate a kdump? I'd like to poke around and see if my assumptions here are correct.. but I believe this crash is being triggered by the lack of the fix for CVE-2024-46697 (https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=f58bab6fd4063913bd8321e99874b8239e9ba726) in the 9.5 kernel. i.e., the specified condition occurs and args.context is set with random junk which, when the kernel attempts to free, sometimes causes a crash -- the effect thereof would be pronounced further on busy servers, probably. |
|
|
In /var/crash i have a lot of kexec-dmesg.log, vmcore, vmcore-dmesg.txt Is that what you would need? |
|
|
https://issues.redhat.com/browse/RHEL-69877 Opened this for now |
|
|
cant see that RHEL issue - despite having a subscription there. |
|
|
Thanks for picking this up, Neil. My instance doesn't run kdump, and is now back in production (and still stable on the previous kernel) - so can't provide further logs I'm afraid. Steve - me neither with a basic sub. Suspect it's a higher level subscription, possibly developer only. |
|
|
Just a short note to say that all kernel-related bug reports become private automatically. This cannot be changed by the submitter. |
|
|
Also having this issue. Server rebooting randomly every few hours. High nfs traffic. New install so the only kernel versions I have are: 5.14.0-503.15.1.el9_5.x86_64 5.14.0-503.14.1.el9_5.x86_64 Both have the same issue. I can send a kdump just let me know where to send. Thanks. |
|
|
@Dale wget https://dl.rockylinux.org/vault/rocky/9.4/BaseOS/x86_64/os/Packages/k/kernel-5.14.0-427.42.1.el9_4.x86_64.rpm wget https://dl.rockylinux.org/vault/rocky/9.4/BaseOS/x86_64/os/Packages/k/kernel-core-5.14.0-427.42.1.el9_4.x86_64.rpm wget https://dl.rockylinux.org/vault/rocky/9.4/BaseOS/x86_64/os/Packages/k/kernel-modules-5.14.0-427.42.1.el9_4.x86_64.rpm wget https://dl.rockylinux.org/vault/rocky/9.4/BaseOS/x86_64/os/Packages/k/kernel-modules-core-5.14.0-427.42.1.el9_4.x86_64.rpm dnf install kernel-5.14.0-427.42.1.el9_4.x86_64.rpm kernel-core-5.14.0-427.42.1.el9_4.x86_64.rpm kernel-modules-5.14.0-427.42.1.el9_4.x86_64.rpm kernel-modules-core-5.14.0-427.42.1.el9_4.x86_64.rpm Try that. I have done that on the servers that where crashing. Since then no more problems. |
|
|
@Steve I've tried this already but with "5.14.0-427.31.1.el9_4.x86_64" since that is the kernel I have still on several other working servers. However it fails to boot and drops to dracut. Just tried "5.14.0-427.42.1.el9_4.x86_64.rpm" and same issue. Our servers need the deprecated megaraid_sas driver. I ran the below to check and the initramfs has the driver: lsinitrd /boot/initramfs-5.14.0-427.31.1.el9_4.x86_64.img | grep mega Fails saying: ata1: SATA link down (SStatus 4 SControl 300) ata2: failed to resume link (SControl 0) ata2: SATA link down (SStatus 4 SControl 0) |
|
|
For clarity, that looks to be the previous kernel, before this bug was apparently introduced. If you're running an established machine (rather than a new one) and have the standard 3-package limit in dnf, then it will still be available in the boot menu and no need to download packages. When we hit problems, we just booted back into the kernel before which amounts to the same thing, and has been totally stable. The issue is only with 503.15.1. No newer kernel, or fix, appears to yet be available. Our available ones: Assumed working as previously used: -rwxr-xr-x. 1 root root 13609800 Oct 16 16:09 vmlinuz-5.14.0-427.40.1.el9_4.x86_64 Works and currently booted. -rwxr-xr-x. 1 root root 13617992 Oct 31 14:13 vmlinuz-5.14.0-427.42.1.el9_4.x86_64 Latest, unstable and crashes within less than an hour. -rwxr-xr-x. 1 root root 14461768 Nov 26 17:37 vmlinuz-5.14.0-503.15.1.el9_5.x86_64 |
|
|
@simon That's all correct. The issue for me is that I did a server cutover to a new machine. The old machine was on Rocky 9.5 but still running the el9_4 kernel so I wasn't aware of the issue while that server was running. The replacement server OS was installed fresh with Rocky 9.5 and so no el9_4 kernels installed. I did install '5.14.0-427.42.1.el9_4.x86_64' manually with: dnf install kernel-5.14.0-427.42.1.el9_4.x86_64.rpm kernel-core-5.14.0-427.42.1.el9_4.x86_64.rpm kernel-modules-5.14.0-427.42.1.el9_4.x86_64.rpm kernel-modules-core-5.14.0-427.42.1.el9_4.x86_64.rpm I also tried copying the kernel, initramfs, and /lib/modules/5.14.0-427.42.1.el9_4.x86_64 from a working server but I still get dropped to a dracut shell with no hard drive found despite the megaraid_sas driver existing in the initramfs. |
|
|
I was able to get our server booted into 5.14.0-427.42.1.el9_4.x86_64. The module at '/lib/modules/5.14.0-427.42.1.el9_4.x86_64/extra/megaraid_sas/megaraid_sas.ko' was missing and was also removed already from our other servers that still have the old 9_4 kernel on them. I was able to get the driver and copy it into: /lib/modules/5.14.0-427.42.1.el9_4.x86_64/extra/megaraid_sas/megaraid_sas.ko then rebuild the initramfs: dracut -f /boot/initramfs-5.14.0-427.42.1.el9_4.x86_64.img 5.14.0-427.42.1.el9_4.x86_64 and this finally booted. Now lsinitrd /boot/initramfs-5.14.0-427.42.1.el9_4.x86_64.img | grep mega looks like this: etc/depmod.d/kmod-megaraid_sas.conf usr/lib/modules/5.14.0-427.42.1.el9_4.x86_64/kernel/drivers/scsi/megaraid usr/lib/modules/5.14.0-427.42.1.el9_4.x86_64/kernel/drivers/scsi/megaraid/megaraid_sas.ko.xz usr/lib/modules/5.14.0-427.42.1.el9_4.x86_64/weak-updates/megaraid_sas usr/lib/modules/5.14.0-427.42.1.el9_4.x86_64/weak-updates/megaraid_sas/megaraid_sas.ko |
|
|
Red Hat released kernel-5.14.0-503.19.1.el9_5 today. This fixes the nfsd bug. I suppose Rocky's kernel will soon become available. |
|
|
Thanks for the update, Akemi, good to know the solution is on its way. |
- Anonymous
