View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0004951Rocky-Linux-8kernelpublic2023-12-06 17:122023-12-06 17:15
ReporterVirgil Fritz Taneza Assigned ToLouis Abel  
PriorityhighSeveritycrashReproducibilityunable to reproduce
Status needinfoResolutionopen 
PlatformRocky Linux 8.5 
Summary0004951: Kernel panic - not syncing: Fatal exception
DescriptionWe are using a web-based NAS (Network-Attached Storage) solution running on top of Rocky Linux that can be deployed in the cloud particularly in Azure and AWS. The NAS solution used ZFS-on-Linux in managing storages which can then be exported via NFS and also shareable via CIFS/SMB. One of our Rocky VM running the NAS solution crashed which seemed to be a hard lockup since the VM stopped logging and the kernel panic logs were only retrieved via the Azure portal. No kernel panic logs found in the VM after it was rebooted. Before the issue was observed, we were running multiple jobs reading/writing data against multiple NFS shares exported by our Rocky-based solution.
 
Below are the kernel panic logs retrieved via the Azure portal:
Here is the stack trace provided by Microsoft, for analysis from Rocky Linux:
 
'***********************' has encountered a fatal error. The guest operating system reported that it failed with the following error codes: ErrorCode0: 0x0, ErrorCode1: 0x0, ErrorCode2: 0x0, ErrorCode3: 0x0, ErrorCode4: 0x0. PreOSId: 0. If the problem persists, contact Product Support for the guest operating system. (Virtual machine ID A2D314F9-4B4B-4C8A-BBDE-************)
 
Guest message:
 
[36431547.130472] Workqueue: nfsd4_callbacks nfsd4_run_cb_work [nfsd]
[36431547.130472] RIP: 0010:__list_add_valid.cold.0+0x26/0x28
[36431547.130472] Code: 00 00 00 c3 48 89 d1 48 c7 c7 20 73 d1 88 48 89 c2 e8 40 ba ca ff 0f 0b 48 89 c1 4c 89 c6 48 c7 c7 78 73 d1 88 e8 2c ba ca ff <0f> 0b 48 89 fe 48 89 c2 48 c7 c7 08 74 d1 88 e8 18 ba ca ff 0f 0b
[36431547.130472] RSP: 0018:ffffa886bfb5fe38 EFLAGS: 00010246
[36431547.130472] RAX: 0000000000000075 RBX: ffff979348fc2688 RCX: 0000000000000000
[36431547.130472] RDX: 0000000000000000 RSI: ffff974f1fdd6858 RDI: ffff974f1fdd6858
[36431547.130472] RBP: ffff975000ee1000 R08: 0000000000000d4b R09: 0000000000000001
[36431547.130472] R10: 0000000000000000 R11: 0000000000000001 R12: ffff979ddcd29948
[36431547.130472] R13: ffff979ddcd29918 R14: ffff975000ee10b8 R15: ffff979ddcd29980
[36431547.130472] FS: 0000000000000000(0000) GS:ffff974f1fdc0000(0000) knlGS:0000000000000000
[36431547.130472] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[36431547.130472] CR2: 000055573197ef44 CR3: 00000004768ce000 CR4: 00000000003506e0
 
[36431547.130472] Call Trace:
[36431547.130472] nfsd4_cb_recall_prepare+0x2b2/0x2f0 [nfsd]
[36431547.130472] nfsd4_run_cb_work+0x3c/0x160 [nfsd]
[36431547.130472] process_one_work+0x1a7/0x360
[36431547.130472] worker_thread+0x30/0x390
[36431547.130472] ? create_worker+0x1a0/0x1a0
[36431547.130472] kthread+0x116/0x130
[36431547.130472] ? kthread_flush_work_fn+0x10/0x10
[36431547.130472] ret_from_fork+0x22/0x40
 
[36431547.130472] Modules linked in: dm_mod rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache nfsd nfs_acl lockd grace ext4 mbcache jbd2 nft_counter xt_conntrack xt_owner nft_compat nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nf_tables_set nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 binfmt_misc ip_set nf_tables nfnetlink vfat fat rpcrdma rdma_ucm ib_srpt ib_isert iscsi_target_mod target_core_mod ib_iser libiscsi scsi_transport_iscsi ib_umad rdma_cm ib_ipoib iw_cm ib_cm intel_rapl_msr zfs(POE) zunicode(POE) zzstd(OE) intel_rapl_common zlua(OE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) crct10dif_pclmul spl(OE) crc32_pclmul ghash_clmulni_intel mlx4_ib ib_uverbs pcspkr i2c_piix4 ib_core hv_utils hv_balloon joydev auth_rpcgss sunrpc btier(OE) xfs libcrc32c mlx4_en mlx4_core nvme nvme_core ata_generic sd_mod t10_pi sg hv_storvsc pci_hyperv hv_netvsc
[36431547.130472] pci_hyperv_intf scsi_transport_fc hid_hyperv hyperv_keyboard hyperv_fb ata_piix libata crc32c_intel hv_vmbus serio_raw
[36431547.453872] ---[ end trace 946a78664b7da3e7 ]---
 
[36431547.459787] RIP: 0010:__list_add_valid.cold.0+0x26/0x28
[36431547.465859] Code: 00 00 00 c3 48 89 d1 48 c7 c7 20 73 d1 88 48 89 c2 e8 40 ba ca ff 0f 0b 48 89 c1 4c 89 c6 48 c7 c7 78 73 d1 88 e8 2c ba ca ff <0f> 0b 48 89 fe 48 89 c2 48 c7 c7 08 74 d1 88 e8 18 ba ca ff 0f 0b
[36431547.490470] RSP: 0018:ffffa886bfb5fe38 EFLAGS: 00010246
[36431547.498210] RAX: 0000000000000075 RBX: ffff979348fc2688 RCX: 0000000000000000
[36431547.508345] RDX: 0000000000000000 RSI: ffff974f1fdd6858 RDI: ffff974f1fdd6858
[36431547.518223] RBP: ffff975000ee1000 R08: 0000000000000d4b R09: 0000000000000001
[36431547.528000] R10: 0000000000000000 R11: 0000000000000001 R12: ffff979ddcd29948
[36431547.538626] R13: ffff979ddcd29918 R14: ffff975000ee10b8 R15: ffff979ddcd29980
[36431547.548692] FS: 0000000000000000(0000) GS:ffff974f1fdc0000(0000) knlGS:0000000000000000
[36431547.559777] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[36431547.568394] CR2: 000055573197ef44 CR3: 00000004768ce000 CR4: 00000000003506e0
[36431547.577925] Kernel panic - not syncing: Fatal exception
[36431547.585801] Kernel Offset: 0x6c00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
 
Below are the relevant package versions used:
1. Rocky version: Rocky Linux release 8.5 (Green Obsidian)
2. nfs-utils: nfs-utils.x86_64 1:2.3.3-46.el8 @baseos
3. Kernel: 4.18.0-348.12.2.el8_5.x86_64
4. ZFSonLinux: 2.1.2-1.el8
Steps To Reproduce1. Deploy Rocky 8.5 on Azure cloud
2. Install ZFS-on-Linux
3. Attach disks to VM
4. Manage attached disks using ZFS to create pools and datasets
5. Export NFSv4 shares
6. Use multiple clients to read/write lots of data from/to the NFS shares
TagsNo tags attached.

Activities

Louis Abel

Louis Abel

2023-12-06 17:15

administrator   ~0005248

Thank you for the report.

Rocky Linux 8.5 is no longer supported. See our version policy for more information: https://wiki.rockylinux.org/rocky/version/

You are recommended to update to 8.9. Once you have done this, you can attempt to reproduce your issue and report back here.

Setting to needinfo.

Issue History

Date Modified Username Field Change
2023-12-06 17:12 Virgil Fritz Taneza New Issue
2023-12-06 17:15 Louis Abel Assigned To => Louis Abel
2023-12-06 17:15 Louis Abel Status new => needinfo
2023-12-06 17:15 Louis Abel Note Added: 0005248