View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0004522Rocky-Linux-8kernelpublic2023-10-19 09:462023-12-01 02:56
ReporterDominika Wanat Assigned To 
PriorityhighSeveritymajorReproducibilityalways
Status newResolutionopen 
Platformel8OSRocky Linux OS Version8.8
Summary0004522: mdraid I/O hang during check/resync
DescriptionWe are experiencing a mdraid thread hang during the check/resync operations on the pools on our HPC storage cluster running on the kernel `4.18.0-477.21.1.el8_8`, which causes the IO to hang. In some cases, we can observe the following trace:

```
Oct 19 01:17:13 tscratch-oss03 kernel: INFO: task md3_raid6:2418 blocked for more than 120 seconds.
Oct 19 01:17:13 tscratch-oss03 kernel: Tainted: G OE X --------- - - 4.18.0-477.21.1.el8_8.x86_64 #1
Oct 19 01:17:13 tscratch-oss03 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 19 01:17:13 tscratch-oss03 kernel: task:md3_raid6 state:D stack: 0 pid: 2418 ppid: 2 flags:0x80004000
Oct 19 01:17:13 tscratch-oss03 kernel: Call Trace:
Oct 19 01:17:13 tscratch-oss03 kernel: __schedule+0x2d1/0x870
Oct 19 01:17:13 tscratch-oss03 kernel: ? raid5_wakeup_stripe_thread+0x98/0x220 [raid456]
Oct 19 01:17:13 tscratch-oss03 kernel: schedule+0x55/0xf0
Oct 19 01:17:13 tscratch-oss03 kernel: raid5d+0x4a7/0x6c0 [raid456]
Oct 19 01:17:13 tscratch-oss03 kernel: ? del_timer_sync+0x25/0x40
Oct 19 01:17:13 tscratch-oss03 kernel: ? finish_wait+0x80/0x80
Oct 19 01:17:13 tscratch-oss03 kernel: ? md_register_thread+0xe0/0xe0
Oct 19 01:17:13 tscratch-oss03 kernel: md_thread+0x94/0x160
Oct 19 01:17:13 tscratch-oss03 kernel: ? finish_wait+0x80/0x80
Oct 19 01:17:13 tscratch-oss03 kernel: kthread+0x134/0x150
Oct 19 01:17:13 tscratch-oss03 kernel: ? set_kthread_struct+0x50/0x50
Oct 19 01:17:13 tscratch-oss03 kernel: ret_from_fork+0x1f/0x40
Oct 19 01:17:13 tscratch-oss03 kernel: INFO: task jbd2/dm-0-8:4036 blocked for more than 120 seconds.
Oct 19 01:17:13 tscratch-oss03 kernel: Tainted: G OE X --------- - - 4.18.0-477.21.1.el8_8.x86_64 #1
Oct 19 01:17:13 tscratch-oss03 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 19 01:17:13 tscratch-oss03 kernel: task:jbd2/dm-0-8 state:D stack: 0 pid: 4036 ppid: 2 flags:0x80004080
Oct 19 01:17:13 tscratch-oss03 kernel: Call Trace:
Oct 19 01:17:13 tscratch-oss03 kernel: __schedule+0x2d1/0x870
Oct 19 01:17:13 tscratch-oss03 kernel: ? __wake_up_common_lock+0x89/0xc0
Oct 19 01:17:13 tscratch-oss03 kernel: ? finish_wait+0x80/0x80
Oct 19 01:17:13 tscratch-oss03 kernel: schedule+0x55/0xf0
Oct 19 01:17:13 tscratch-oss03 kernel: jbd2_journal_commit_transaction+0x259/0x19f0 [jbd2]
Oct 19 01:17:13 tscratch-oss03 kernel: ? newidle_balance+0x279/0x3c0
Oct 19 01:17:13 tscratch-oss03 kernel: ? finish_wait+0x80/0x80
Oct 19 01:17:13 tscratch-oss03 kernel: ? finish_task_switch+0x86/0x2e0
Oct 19 01:17:13 tscratch-oss03 kernel: ? lock_timer_base+0x67/0x90
Oct 19 01:17:13 tscratch-oss03 kernel: kjournald2+0xbd/0x270 [jbd2]
Oct 19 01:17:13 tscratch-oss03 kernel: ? finish_wait+0x80/0x80
Oct 19 01:17:13 tscratch-oss03 kernel: ? commit_timeout+0x10/0x10 [jbd2]
Oct 19 01:17:13 tscratch-oss03 kernel: kthread+0x134/0x150
Oct 19 01:17:13 tscratch-oss03 kernel: ? set_kthread_struct+0x50/0x50
Oct 19 01:17:13 tscratch-oss03 kernel: ret_from_fork+0x1f/0x40
Oct 19 01:17:13 tscratch-oss03 kernel: INFO: task jbd2/dm-1-8:4037 blocked for more than 120 seconds.
Oct 19 01:17:13 tscratch-oss03 kernel: Tainted: G OE X --------- - - 4.18.0-477.21.1.el8_8.x86_64 #1
Oct 19 01:17:13 tscratch-oss03 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 19 01:17:13 tscratch-oss03 kernel: task:jbd2/dm-1-8 state:D stack: 0 pid: 4037 ppid: 2 flags:0x80004080
Oct 19 01:17:13 tscratch-oss03 kernel: Call Trace:
Oct 19 01:17:13 tscratch-oss03 kernel: __schedule+0x2d1/0x870
Oct 19 01:17:13 tscratch-oss03 kernel: schedule+0x55/0xf0
Oct 19 01:17:13 tscratch-oss03 kernel: md_write_start+0x14f/0x220
Oct 19 01:17:13 tscratch-oss03 kernel: ? finish_wait+0x80/0x80
Oct 19 01:17:13 tscratch-oss03 kernel: raid5_make_request+0x93/0xe00 [raid456]
Oct 19 01:17:13 tscratch-oss03 kernel: ? finish_wait+0x80/0x80
Oct 19 01:17:13 tscratch-oss03 kernel: ? mempool_alloc+0x67/0x180
Oct 19 01:17:13 tscratch-oss03 kernel: ? do_wait_intr_irq+0xa0/0xa0
ct 19 01:17:13 tscratch-oss03 kernel: ? generic_make_request_checks+0x2a5/0x4f0
Oct 19 01:17:13 tscratch-oss03 kernel: ? dm_io_acct+0x49/0xe0 [dm_mod]
Oct 19 01:17:13 tscratch-oss03 kernel: ? finish_wait+0x80/0x80
Oct 19 01:17:13 tscratch-oss03 kernel: md_handle_request+0x128/0x1b0
Oct 19 01:17:13 tscratch-oss03 kernel: md_make_request+0x5b/0xb0
Oct 19 01:17:13 tscratch-oss03 kernel: generic_make_request_no_check+0x202/0x330
Oct 19 01:17:13 tscratch-oss03 kernel: submit_bio+0x3c/0x160
Oct 19 01:17:13 tscratch-oss03 kernel: ? bio_add_page+0x46/0x60
Oct 19 01:17:13 tscratch-oss03 kernel: submit_bh_wbc+0x16a/0x1a0
Oct 19 01:17:13 tscratch-oss03 kernel: jbd2_journal_commit_transaction+0x6b0/0x19f0 [jbd2]
Oct 19 01:17:13 tscratch-oss03 kernel: ? finish_task_switch+0x86/0x2e0
Oct 19 01:17:13 tscratch-oss03 kernel: kjournald2+0xbd/0x270 [jbd2]
Oct 19 01:17:13 tscratch-oss03 kernel: ? finish_wait+0x80/0x80
Oct 19 01:17:13 tscratch-oss03 kernel: ? commit_timeout+0x10/0x10 [jbd2]
Oct 19 01:17:13 tscratch-oss03 kernel: kthread+0x134/0x150
Oct 19 01:17:13 tscratch-oss03 kernel: ? set_kthread_struct+0x50/0x50
Oct 19 01:17:13 tscratch-oss03 kernel: ret_from_fork+0x1f/0x40
Oct 19 01:17:13 tscratch-oss03 kernel: INFO: task ll_ost02_000:4110 blocked for more than 120 seconds.
Oct 19 01:17:13 tscratch-oss03 kernel: Tainted: G OE X --------- - - 4.18.0-477.21.1.el8_8.x86_64 #1
Oct 19 01:17:13 tscratch-oss03 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
```

In some cases, this stack trace is not generated, and the problem is escalated to the upper storage layer, where Lustre informs us that it is unhealthy because of the RPC timeouts. We are confident that the mdraid layer causes the problem because we found a similar bug report on Ubuntu: https://bugs.launchpad.net/ubuntu/+source/linux-signed-hwe-5.11/+bug/1942935 and the workaround presented there: https://bugs.launchpad.net/ubuntu/+source/linux-signed-hwe-5.11/+bug/1942935/comments/10 unlocks the IO on the affected pool and the Lustre restores to a healthy state. Unfortunately, applications on the HPC cluster are IO-sensitive, so even this solution causes many timeouts and crashed jobs. The above-mentioned bug report also has a reproducer and the patch proposition, which was never merged into the kernel mainline.
Tagsio hang, kernel, mdraid

Activities

Perry MacDonald

Perry MacDonald

2023-11-24 04:02

reporter   ~0005203

I think I might be running into a similar situation, though ours is with storage for a handful of KVM VMs. I've attached a copy of the traces I'm seeing in this similar situation.

It appears to have started after upgrading to 8.8, we have 7 of these boxes in production currently but only two have been upgraded to 8.8. The others are running on 8.7 and have been stable.

Hardware is a mix of Dell R7515 and R7525's with NVIDIA GPUs running 3-4 Windows VMs each as a makeshift VDI solution. Until now they have bee fairly reliable.

Storage is 4x 2T 970 EVO Plus NVMe drives in a md raid 5 with xfs on top. Normally I would not run a config like this but latency/performance is the goal and these workstations can fairly easily be rebuilt.

So far I've caught this issue once while a raid check was occuring, but I don't know for sure on the others. Currently this happens every few days to two weeks.

When this hits, the VMs appear to lose access to their storage. The mount is still there and I can see the VM files, but as with the linked reports any attempts to interact with the array will hang. Hard reboot of the box is the only way out of it.

So far I've tested adding nvme_core.default_ps_max_latency_us=0 after looking at some ASPM issue posts, but no change. I've also tweaked the group_thread_cnt to 6 and the stripe_cache_size to 8192. Seems to have improved performance, but still having this crash issue.

Happy to provide more details if needed, been ongoing for a couple of months now so I may have missed some details.
redacted-trace-kvm.txt (25,465 bytes)    
Nov 23 16:22:57 HOSTNAME kernel: INFO: task worker:964112 blocked for more than 120 seconds.
Nov 23 16:22:57 HOSTNAME kernel:      Tainted: P           OE    --------- -  - 4.18.0-477.21.1.el8_8.x86_64 #1
Nov 23 16:22:57 HOSTNAME kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 23 16:22:57 HOSTNAME kernel: task:worker          state:D stack:    0 pid:964112 ppid:     1 flags:0x00004180
Nov 23 16:22:57 HOSTNAME kernel: Call Trace:
Nov 23 16:22:57 HOSTNAME kernel: __schedule+0x2d1/0x870
Nov 23 16:22:57 HOSTNAME kernel: ? md_bitmap_get_counter+0x42/0xe0
Nov 23 16:22:57 HOSTNAME kernel: schedule+0x55/0xf0
Nov 23 16:22:57 HOSTNAME kernel: raid5_get_active_stripe+0x21e/0x4e0 [raid456]
Nov 23 16:22:57 HOSTNAME kernel: ? finish_wait+0x80/0x80
Nov 23 16:22:57 HOSTNAME kernel: raid5_make_request+0x20c/0xe00 [raid456]
Nov 23 16:22:57 HOSTNAME kernel: ? do_wait_intr_irq+0xa0/0xa0
Nov 23 16:22:57 HOSTNAME kernel: ? generic_make_request_checks+0x2a5/0x4f0
Nov 23 16:22:57 HOSTNAME kernel: ? dm_io_acct+0x49/0xe0 [dm_mod]
Nov 23 16:22:57 HOSTNAME kernel: ? finish_wait+0x80/0x80
Nov 23 16:22:57 HOSTNAME kernel: md_handle_request+0x128/0x1b0
Nov 23 16:22:57 HOSTNAME kernel: md_make_request+0x5b/0xb0
Nov 23 16:22:57 HOSTNAME kernel: generic_make_request_no_check+0x202/0x330
Nov 23 16:22:57 HOSTNAME kernel: submit_bio+0x3c/0x160
Nov 23 16:22:57 HOSTNAME kernel: iomap_submit_ioend.isra.42+0x4a/0x80
Nov 23 16:22:57 HOSTNAME kernel: iomap_writepage_map+0x41a/0x670
Nov 23 16:22:57 HOSTNAME kernel: write_cache_pages+0x197/0x420
Nov 23 16:22:57 HOSTNAME kernel: ? iomap_write_end+0x210/0x210
Nov 23 16:22:57 HOSTNAME kernel: iomap_writepages+0x1c/0x40
Nov 23 16:22:57 HOSTNAME kernel: xfs_vm_writepages+0x7e/0xb0 [xfs]
Nov 23 16:22:57 HOSTNAME kernel: do_writepages+0xc6/0x1d0
Nov 23 16:22:57 HOSTNAME kernel: __filemap_fdatawrite_range+0xcb/0x100
Nov 23 16:22:57 HOSTNAME kernel: file_write_and_wait_range+0x4c/0xa0
Nov 23 16:22:57 HOSTNAME kernel: xfs_file_fsync+0x5d/0x210 [xfs]
Nov 23 16:22:57 HOSTNAME kernel: do_fsync+0x38/0x70
Nov 23 16:22:57 HOSTNAME kernel: __x64_sys_fdatasync+0x13/0x20
Nov 23 16:22:57 HOSTNAME kernel: do_syscall_64+0x5b/0x1b0
Nov 23 16:22:57 HOSTNAME kernel: entry_SYSCALL_64_after_hwframe+0x61/0xc6
Nov 23 16:22:57 HOSTNAME kernel: RIP: 0033:0x7f1e1b5d68af
Nov 23 16:22:57 HOSTNAME kernel: Code: Unable to access opcode bytes at RIP 0x7f1e1b5d6885.
Nov 23 16:22:57 HOSTNAME kernel: RSP: 002b:00007f1e16b88760 EFLAGS: 00000293 ORIG_RAX: 000000000000004b
Nov 23 16:22:57 HOSTNAME kernel: RAX: ffffffffffffffda RBX: 000000000000000e RCX: 00007f1e1b5d68af
Nov 23 16:22:57 HOSTNAME kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 000000000000000e
Nov 23 16:22:57 HOSTNAME kernel: RBP: 0000560dfd7b1f00 R08: 0000000000000000 R09: 00000000ffffffff
Nov 23 16:22:57 HOSTNAME kernel: R10: 00007f1e16b88760 R11: 0000000000000293 R12: 0000560dfb35fe46
Nov 23 16:22:57 HOSTNAME kernel: R13: 0000560dfd5fd4b8 R14: 0000560dff4755a0 R15: 00007f1e16b888c0
Nov 23 16:22:57 HOSTNAME kernel: INFO: task worker:967193 blocked for more than 120 seconds.
Nov 23 16:22:57 HOSTNAME kernel:      Tainted: P           OE    --------- -  - 4.18.0-477.21.1.el8_8.x86_64 #1
Nov 23 16:22:57 HOSTNAME kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 23 16:22:57 HOSTNAME kernel: task:worker          state:D stack:    0 pid:967193 ppid:     1 flags:0x00000180
Nov 23 16:22:57 HOSTNAME kernel: Call Trace:
Nov 23 16:22:57 HOSTNAME kernel: __schedule+0x2d1/0x870
Nov 23 16:22:57 HOSTNAME kernel: ? xfs_trans_add_item+0x33/0xb0 [xfs]
Nov 23 16:22:57 HOSTNAME kernel: schedule+0x55/0xf0
Nov 23 16:22:57 HOSTNAME kernel: raid5_get_active_stripe+0x21e/0x4e0 [raid456]
Nov 23 16:22:57 HOSTNAME kernel: ? finish_wait+0x80/0x80
Nov 23 16:22:57 HOSTNAME kernel: raid5_make_request+0x20c/0xe00 [raid456]
Nov 23 16:22:57 HOSTNAME kernel: ? do_wait_intr_irq+0xa0/0xa0
Nov 23 16:22:57 HOSTNAME kernel: ? generic_make_request_checks+0x2a5/0x4f0
Nov 23 16:22:57 HOSTNAME kernel: ? dm_io_acct+0x49/0xe0 [dm_mod]
Nov 23 16:22:57 HOSTNAME kernel: ? finish_wait+0x80/0x80
Nov 23 16:22:57 HOSTNAME kernel: md_handle_request+0x128/0x1b0
Nov 23 16:22:57 HOSTNAME kernel: md_make_request+0x5b/0xb0
Nov 23 16:22:57 HOSTNAME kernel: generic_make_request_no_check+0x202/0x330
Nov 23 16:22:57 HOSTNAME kernel: submit_bio+0x3c/0x160
Nov 23 16:22:57 HOSTNAME kernel: iomap_submit_ioend.isra.42+0x4a/0x80
Nov 23 16:22:57 HOSTNAME kernel: iomap_writepage_map+0x41a/0x670
Nov 23 16:22:57 HOSTNAME kernel: write_cache_pages+0x197/0x420
Nov 23 16:22:57 HOSTNAME kernel: ? iomap_write_end+0x210/0x210
Nov 23 16:22:57 HOSTNAME kernel: iomap_writepages+0x1c/0x40
Nov 23 16:22:57 HOSTNAME kernel: xfs_vm_writepages+0x7e/0xb0 [xfs]
Nov 23 16:22:57 HOSTNAME kernel: do_writepages+0xc6/0x1d0
Nov 23 16:22:57 HOSTNAME kernel: __filemap_fdatawrite_range+0xcb/0x100
Nov 23 16:22:57 HOSTNAME kernel: file_write_and_wait_range+0x4c/0xa0
Nov 23 16:22:57 HOSTNAME kernel: xfs_file_fsync+0x5d/0x210 [xfs]
Nov 23 16:22:57 HOSTNAME kernel: do_fsync+0x38/0x70
Nov 23 16:22:57 HOSTNAME kernel: __x64_sys_fdatasync+0x13/0x20
Nov 23 16:22:57 HOSTNAME kernel: do_syscall_64+0x5b/0x1b0
Nov 23 16:22:57 HOSTNAME kernel: entry_SYSCALL_64_after_hwframe+0x61/0xc6
Nov 23 16:22:57 HOSTNAME kernel: RIP: 0033:0x7f1e1b5d68af
Nov 23 16:22:57 HOSTNAME kernel: Code: Unable to access opcode bytes at RIP 0x7f1e1b5d6885.
Nov 23 16:22:57 HOSTNAME kernel: RSP: 002b:00007efd8b7fd760 EFLAGS: 00000293 ORIG_RAX: 000000000000004b
Nov 23 16:22:57 HOSTNAME kernel: RAX: ffffffffffffffda RBX: 000000000000000f RCX: 00007f1e1b5d68af
Nov 23 16:22:57 HOSTNAME kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 000000000000000f
Nov 23 16:22:57 HOSTNAME kernel: RBP: 0000560dfd7ba6b0 R08: 0000000000000000 R09: 00000000ffffffff
Nov 23 16:22:57 HOSTNAME kernel: R10: 00007efd8b7fd760 R11: 0000000000000293 R12: 0000560dfb35fe46
Nov 23 16:22:57 HOSTNAME kernel: R13: 0000560dfd5fd4b8 R14: 0000560dff68df20 R15: 00007efd8b7fd8c0
Nov 23 16:22:57 HOSTNAME kernel: INFO: task worker:958057 blocked for more than 120 seconds.
Nov 23 16:22:57 HOSTNAME kernel:      Tainted: P           OE    --------- -  - 4.18.0-477.21.1.el8_8.x86_64 #1
Nov 23 16:22:57 HOSTNAME kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 23 16:22:57 HOSTNAME kernel: task:worker          state:D stack:    0 pid:958057 ppid:     1 flags:0x00000180
Nov 23 16:22:57 HOSTNAME kernel: Call Trace:
Nov 23 16:22:57 HOSTNAME kernel: __schedule+0x2d1/0x870
Nov 23 16:22:57 HOSTNAME kernel: ? xfs_trans_add_item+0x33/0xb0 [xfs]
Nov 23 16:22:57 HOSTNAME kernel: schedule+0x55/0xf0
Nov 23 16:22:57 HOSTNAME kernel: raid5_get_active_stripe+0x21e/0x4e0 [raid456]
Nov 23 16:22:57 HOSTNAME kernel: ? finish_wait+0x80/0x80
Nov 23 16:22:57 HOSTNAME kernel: raid5_make_request+0x20c/0xe00 [raid456]
Nov 23 16:22:57 HOSTNAME kernel: ? do_wait_intr_irq+0xa0/0xa0
Nov 23 16:22:57 HOSTNAME kernel: ? generic_make_request_checks+0x2a5/0x4f0
Nov 23 16:22:57 HOSTNAME kernel: ? dm_io_acct+0x49/0xe0 [dm_mod]
Nov 23 16:22:57 HOSTNAME kernel: ? finish_wait+0x80/0x80
Nov 23 16:22:57 HOSTNAME kernel: md_handle_request+0x128/0x1b0
Nov 23 16:22:57 HOSTNAME kernel: md_make_request+0x5b/0xb0
Nov 23 16:22:57 HOSTNAME kernel: generic_make_request_no_check+0x202/0x330
Nov 23 16:22:57 HOSTNAME kernel: submit_bio+0x3c/0x160
Nov 23 16:22:57 HOSTNAME kernel: iomap_submit_ioend.isra.42+0x4a/0x80
Nov 23 16:22:57 HOSTNAME kernel: iomap_writepage_map+0x41a/0x670
Nov 23 16:22:57 HOSTNAME kernel: write_cache_pages+0x197/0x420
Nov 23 16:22:57 HOSTNAME kernel: ? iomap_write_end+0x210/0x210
Nov 23 16:22:57 HOSTNAME kernel: iomap_writepages+0x1c/0x40
Nov 23 16:22:57 HOSTNAME kernel: xfs_vm_writepages+0x7e/0xb0 [xfs]
Nov 23 16:22:57 HOSTNAME kernel: do_writepages+0xc6/0x1d0
Nov 23 16:22:57 HOSTNAME kernel: __filemap_fdatawrite_range+0xcb/0x100
Nov 23 16:22:57 HOSTNAME kernel: file_write_and_wait_range+0x4c/0xa0
Nov 23 16:22:57 HOSTNAME kernel: xfs_file_fsync+0x5d/0x210 [xfs]
Nov 23 16:22:57 HOSTNAME kernel: do_fsync+0x38/0x70
Nov 23 16:22:57 HOSTNAME kernel: __x64_sys_fdatasync+0x13/0x20
Nov 23 16:22:57 HOSTNAME kernel: do_syscall_64+0x5b/0x1b0
Nov 23 16:22:57 HOSTNAME kernel: entry_SYSCALL_64_after_hwframe+0x61/0xc6
Nov 23 16:22:57 HOSTNAME kernel: RIP: 0033:0x7f85b14938af
Nov 23 16:22:57 HOSTNAME kernel: Code: Unable to access opcode bytes at RIP 0x7f85b1493885.
Nov 23 16:22:57 HOSTNAME kernel: RSP: 002b:00007f65305c4760 EFLAGS: 00000293 ORIG_RAX: 000000000000004b
Nov 23 16:22:57 HOSTNAME kernel: RAX: ffffffffffffffda RBX: 000000000000000f RCX: 00007f85b14938af
Nov 23 16:22:57 HOSTNAME kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 000000000000000f
Nov 23 16:22:57 HOSTNAME kernel: RBP: 0000560419bae6b0 R08: 0000000000000000 R09: 00000000ffffffff
Nov 23 16:22:57 HOSTNAME kernel: R10: 00007f65305c4760 R11: 0000000000000293 R12: 0000560418c88e46
Nov 23 16:22:57 HOSTNAME kernel: R13: 00005604199f14b8 R14: 000056041b9bd4a0 R15: 00007f65305c48c0
Nov 23 16:22:57 HOSTNAME kernel: INFO: task worker:961464 blocked for more than 120 seconds.
Nov 23 16:22:57 HOSTNAME kernel:      Tainted: P           OE    --------- -  - 4.18.0-477.21.1.el8_8.x86_64 #1
Nov 23 16:22:57 HOSTNAME kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 23 16:22:57 HOSTNAME kernel: task:worker          state:D stack:    0 pid:961464 ppid:     1 flags:0x00000180
Nov 23 16:22:57 HOSTNAME kernel: Call Trace:
Nov 23 16:22:57 HOSTNAME kernel: __schedule+0x2d1/0x870
Nov 23 16:22:57 HOSTNAME kernel: schedule+0x55/0xf0
Nov 23 16:22:57 HOSTNAME kernel: raid5_get_active_stripe+0x21e/0x4e0 [raid456]
Nov 23 16:22:57 HOSTNAME kernel: ? finish_wait+0x80/0x80
Nov 23 16:22:57 HOSTNAME kernel: raid5_make_request+0x20c/0xe00 [raid456]
Nov 23 16:22:57 HOSTNAME kernel: ? do_wait_intr_irq+0xa0/0xa0
Nov 23 16:22:57 HOSTNAME kernel: ? generic_make_request_checks+0x2a5/0x4f0
Nov 23 16:22:57 HOSTNAME kernel: ? dm_io_acct+0x49/0xe0 [dm_mod]
Nov 23 16:22:57 HOSTNAME kernel: ? finish_wait+0x80/0x80
Nov 23 16:22:57 HOSTNAME kernel: md_handle_request+0x128/0x1b0
Nov 23 16:22:57 HOSTNAME kernel: md_make_request+0x5b/0xb0
Nov 23 16:22:57 HOSTNAME kernel: generic_make_request_no_check+0x202/0x330
Nov 23 16:22:57 HOSTNAME kernel: submit_bio+0x3c/0x160
Nov 23 16:22:57 HOSTNAME kernel: iomap_submit_ioend.isra.42+0x4a/0x80
Nov 23 16:22:57 HOSTNAME kernel: iomap_writepage_map+0x41a/0x670
Nov 23 16:22:57 HOSTNAME kernel: write_cache_pages+0x197/0x420
Nov 23 16:22:57 HOSTNAME kernel: ? iomap_write_end+0x210/0x210
Nov 23 16:22:57 HOSTNAME kernel: iomap_writepages+0x1c/0x40
Nov 23 16:22:57 HOSTNAME kernel: xfs_vm_writepages+0x7e/0xb0 [xfs]
Nov 23 16:22:57 HOSTNAME kernel: do_writepages+0xc6/0x1d0
Nov 23 16:22:57 HOSTNAME kernel: __filemap_fdatawrite_range+0xcb/0x100
Nov 23 16:22:57 HOSTNAME kernel: file_write_and_wait_range+0x4c/0xa0
Nov 23 16:22:57 HOSTNAME kernel: xfs_file_fsync+0x5d/0x210 [xfs]
Nov 23 16:22:57 HOSTNAME kernel: do_fsync+0x38/0x70
Nov 23 16:22:57 HOSTNAME kernel: __x64_sys_fdatasync+0x13/0x20
Nov 23 16:22:57 HOSTNAME kernel: do_syscall_64+0x5b/0x1b0
Nov 23 16:22:57 HOSTNAME kernel: entry_SYSCALL_64_after_hwframe+0x61/0xc6
Nov 23 16:22:57 HOSTNAME kernel: RIP: 0033:0x7f85b14938af
Nov 23 16:22:57 HOSTNAME kernel: Code: Unable to access opcode bytes at RIP 0x7f85b1493885.
Nov 23 16:22:57 HOSTNAME kernel: RSP: 002b:00007f650d8f3760 EFLAGS: 00000293 ORIG_RAX: 000000000000004b
Nov 23 16:22:57 HOSTNAME kernel: RAX: ffffffffffffffda RBX: 000000000000000e RCX: 00007f85b14938af
Nov 23 16:22:57 HOSTNAME kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 000000000000000e
Nov 23 16:22:57 HOSTNAME kernel: RBP: 0000560419ba5f00 R08: 0000000000000000 R09: 00000000ffffffff
Nov 23 16:22:57 HOSTNAME kernel: R10: 00007f650d8f3760 R11: 0000000000000293 R12: 0000560418c88e46
Nov 23 16:22:57 HOSTNAME kernel: R13: 00005604199f14b8 R14: 000056041b909ed0 R15: 00007f650d8f38c0
Nov 23 16:22:57 HOSTNAME kernel: INFO: task worker:968890 blocked for more than 120 seconds.
Nov 23 16:22:57 HOSTNAME kernel:      Tainted: P           OE    --------- -  - 4.18.0-477.21.1.el8_8.x86_64 #1
Nov 23 16:22:57 HOSTNAME kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 23 16:22:57 HOSTNAME kernel: task:worker          state:D stack:    0 pid:968890 ppid:     1 flags:0x00000180
Nov 23 16:22:57 HOSTNAME kernel: Call Trace:
Nov 23 16:22:57 HOSTNAME kernel: __schedule+0x2d1/0x870
Nov 23 16:22:57 HOSTNAME kernel: schedule+0x55/0xf0
Nov 23 16:22:57 HOSTNAME kernel: raid5_get_active_stripe+0x21e/0x4e0 [raid456]
Nov 23 16:22:57 HOSTNAME kernel: ? finish_wait+0x80/0x80
Nov 23 16:22:57 HOSTNAME kernel: raid5_make_request+0x20c/0xe00 [raid456]
Nov 23 16:22:57 HOSTNAME kernel: ? do_wait_intr_irq+0xa0/0xa0
Nov 23 16:22:57 HOSTNAME kernel: ? generic_make_request_checks+0x2a5/0x4f0
Nov 23 16:22:57 HOSTNAME kernel: ? dm_io_acct+0x49/0xe0 [dm_mod]
Nov 23 16:22:57 HOSTNAME kernel: ? finish_wait+0x80/0x80
Nov 23 16:22:57 HOSTNAME kernel: md_handle_request+0x128/0x1b0
Nov 23 16:22:57 HOSTNAME kernel: md_make_request+0x5b/0xb0
Nov 23 16:22:57 HOSTNAME kernel: generic_make_request_no_check+0x202/0x330
Nov 23 16:22:57 HOSTNAME kernel: submit_bio+0x3c/0x160
Nov 23 16:22:57 HOSTNAME kernel: ? timerqueue_del+0x2a/0x50
Nov 23 16:22:57 HOSTNAME kernel: iomap_submit_ioend.isra.42+0x4a/0x80
Nov 23 16:22:57 HOSTNAME kernel: xfs_vm_writepages+0x7e/0xb0 [xfs]
Nov 23 16:22:57 HOSTNAME kernel: do_writepages+0xc6/0x1d0
Nov 23 16:22:57 HOSTNAME kernel: __filemap_fdatawrite_range+0xcb/0x100
Nov 23 16:22:57 HOSTNAME kernel: file_write_and_wait_range+0x4c/0xa0
Nov 23 16:22:57 HOSTNAME kernel: xfs_file_fsync+0x5d/0x210 [xfs]
Nov 23 16:22:57 HOSTNAME kernel: do_fsync+0x38/0x70
Nov 23 16:22:57 HOSTNAME kernel: __x64_sys_fdatasync+0x13/0x20
Nov 23 16:22:57 HOSTNAME kernel: do_syscall_64+0x5b/0x1b0
Nov 23 16:22:57 HOSTNAME kernel: entry_SYSCALL_64_after_hwframe+0x61/0xc6
Nov 23 16:22:57 HOSTNAME kernel: RIP: 0033:0x7fc655b208af
Nov 23 16:22:57 HOSTNAME kernel: Code: Unable to access opcode bytes at RIP 0x7fc655b20885.
Nov 23 16:22:57 HOSTNAME kernel: RSP: 002b:00007fa5d3ffe760 EFLAGS: 00000293 ORIG_RAX: 000000000000004b
Nov 23 16:22:57 HOSTNAME kernel: RAX: ffffffffffffffda RBX: 000000000000000e RCX: 00007fc655b208af
Nov 23 16:22:57 HOSTNAME kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 000000000000000e
Nov 23 16:22:57 HOSTNAME kernel: RBP: 0000556ca2528f00 R08: 0000000000000000 R09: 00000000ffffffff
Nov 23 16:22:57 HOSTNAME kernel: R10: 00007fa5d3ffe760 R11: 0000000000000293 R12: 0000556ca15b5e46
Nov 23 16:22:57 HOSTNAME kernel: R13: 0000556ca23744b8 R14: 0000556ca44a8d60 R15: 00007fa5d3ffe8c0
Nov 23 16:22:57 HOSTNAME kernel: INFO: task worker:968837 blocked for more than 120 seconds.
Nov 23 16:22:57 HOSTNAME kernel:      Tainted: P           OE    --------- -  - 4.18.0-477.21.1.el8_8.x86_64 #1
Nov 23 16:22:57 HOSTNAME kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 23 16:22:57 HOSTNAME kernel: task:worker          state:D stack:    0 pid:968837 ppid:     1 flags:0x00000180
Nov 23 16:22:57 HOSTNAME kernel: Call Trace:
Nov 23 16:22:57 HOSTNAME kernel: __schedule+0x2d1/0x870
Nov 23 16:22:57 HOSTNAME kernel: schedule+0x55/0xf0
Nov 23 16:22:57 HOSTNAME kernel: raid5_get_active_stripe+0x21e/0x4e0 [raid456]
Nov 23 16:22:57 HOSTNAME kernel: ? finish_wait+0x80/0x80
Nov 23 16:22:57 HOSTNAME kernel: raid5_make_request+0x20c/0xe00 [raid456]
Nov 23 16:22:57 HOSTNAME kernel: ? do_wait_intr_irq+0xa0/0xa0
Nov 23 16:22:57 HOSTNAME kernel: ? generic_make_request_checks+0x2a5/0x4f0
Nov 23 16:22:57 HOSTNAME kernel: ? dm_io_acct+0x49/0xe0 [dm_mod]
Nov 23 16:22:57 HOSTNAME kernel: ? finish_wait+0x80/0x80
Nov 23 16:22:57 HOSTNAME kernel: md_handle_request+0x128/0x1b0
Nov 23 16:22:57 HOSTNAME kernel: md_make_request+0x5b/0xb0
Nov 23 16:22:57 HOSTNAME kernel: generic_make_request_no_check+0x202/0x330
Nov 23 16:22:57 HOSTNAME kernel: submit_bio+0x3c/0x160
Nov 23 16:22:57 HOSTNAME kernel: iomap_submit_ioend.isra.42+0x4a/0x80
Nov 23 16:22:57 HOSTNAME kernel: iomap_writepage_map+0x41a/0x670
Nov 23 16:22:57 HOSTNAME kernel: write_cache_pages+0x197/0x420
Nov 23 16:22:57 HOSTNAME kernel: ? iomap_write_end+0x210/0x210
Nov 23 16:22:57 HOSTNAME kernel: ? __schedule+0x2d9/0x870
Nov 23 16:22:57 HOSTNAME kernel: ? hrtimer_start_range_ns+0x11b/0x310
Nov 23 16:22:57 HOSTNAME kernel: ? timerqueue_del+0x2a/0x50
Nov 23 16:22:57 HOSTNAME kernel: iomap_writepages+0x1c/0x40
Nov 23 16:22:57 HOSTNAME kernel: xfs_vm_writepages+0x7e/0xb0 [xfs]
Nov 23 16:22:57 HOSTNAME kernel: do_writepages+0xc6/0x1d0
Nov 23 16:22:57 HOSTNAME kernel: __filemap_fdatawrite_range+0xcb/0x100
Nov 23 16:22:57 HOSTNAME kernel: file_write_and_wait_range+0x4c/0xa0
Nov 23 16:22:57 HOSTNAME kernel: xfs_file_fsync+0x5d/0x210 [xfs]
Nov 23 16:22:57 HOSTNAME kernel: do_fsync+0x38/0x70
Nov 23 16:22:57 HOSTNAME kernel: __x64_sys_fdatasync+0x13/0x20
Nov 23 16:22:57 HOSTNAME kernel: do_syscall_64+0x5b/0x1b0
Nov 23 16:22:57 HOSTNAME kernel: entry_SYSCALL_64_after_hwframe+0x61/0xc6
Nov 23 16:22:57 HOSTNAME kernel: RIP: 0033:0x7fded47178af
Nov 23 16:22:57 HOSTNAME kernel: Code: Unable to access opcode bytes at RIP 0x7fded4717885.
Nov 23 16:22:57 HOSTNAME kernel: RSP: 002b:00007fdecfcc9760 EFLAGS: 00000293 ORIG_RAX: 000000000000004b
Nov 23 16:22:57 HOSTNAME kernel: RAX: ffffffffffffffda RBX: 000000000000000e RCX: 00007fded47178af
Nov 23 16:22:57 HOSTNAME kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 000000000000000e
Nov 23 16:22:57 HOSTNAME kernel: RBP: 0000560f9a5df440 R08: 0000000000000000 R09: 00000000ffffffff
Nov 23 16:22:57 HOSTNAME kernel: R10: 00007fdecfcc9760 R11: 0000000000000293 R12: 0000560f98ae9e46
Nov 23 16:22:57 HOSTNAME kernel: R13: 0000560f9a42a358 R14: 0000560f9cdd7700 R15: 00007fdecfcc98c0
Nov 23 16:22:57 HOSTNAME kernel: INFO: task kworker/u258:4:947247 blocked for more than 120 seconds.
Nov 23 16:22:57 HOSTNAME kernel:      Tainted: P           OE    --------- -  - 4.18.0-477.21.1.el8_8.x86_64 #1
Nov 23 16:22:57 HOSTNAME kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 23 16:22:57 HOSTNAME kernel: task:kworker/u258:4  state:D stack:    0 pid:947247 ppid:     2 flags:0x80004080
Nov 23 16:22:57 HOSTNAME kernel: Workqueue: xfs-cil/dm-2 xlog_cil_push_work [xfs]
Nov 23 16:22:57 HOSTNAME kernel: Call Trace:
Nov 23 16:22:57 HOSTNAME kernel: __schedule+0x2d1/0x870
Nov 23 16:22:57 HOSTNAME kernel: schedule+0x55/0xf0
Nov 23 16:22:57 HOSTNAME kernel: xlog_state_get_iclog_space+0x10b/0x340 [xfs]
Nov 23 16:22:57 HOSTNAME kernel: ? xlog_write+0x5ac/0x740 [xfs]
Nov 23 16:22:57 HOSTNAME kernel: ? wake_up_q+0x70/0x70
Nov 23 16:22:57 HOSTNAME kernel: xlog_write+0x152/0x740 [xfs]
Nov 23 16:22:57 HOSTNAME kernel: xlog_cil_push_work+0x2ab/0x550 [xfs]
Nov 23 16:22:57 HOSTNAME kernel: ? __switch_to_asm+0x43/0x80
Nov 23 16:22:57 HOSTNAME kernel: process_one_work+0x1a7/0x360
Nov 23 16:22:57 HOSTNAME kernel: worker_thread+0x1ce/0x390
Nov 23 16:22:57 HOSTNAME kernel: ? create_worker+0x1a0/0x1a0
Nov 23 16:22:57 HOSTNAME kernel: kthread+0x134/0x150
Nov 23 16:22:57 HOSTNAME kernel: ? set_kthread_struct+0x50/0x50
Nov 23 16:22:57 HOSTNAME kernel: ret_from_fork+0x35/0x40
Nov 23 16:22:57 HOSTNAME kernel: INFO: task kworker/u258:7:961929 blocked for more than 120 seconds.
Nov 23 16:22:57 HOSTNAME kernel:      Tainted: P           OE    --------- -  - 4.18.0-477.21.1.el8_8.x86_64 #1
Nov 23 16:22:57 HOSTNAME kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 23 16:22:57 HOSTNAME kernel: task:kworker/u258:7  state:D stack:    0 pid:961929 ppid:     2 flags:0x80004080
Nov 23 16:22:57 HOSTNAME kernel: Workqueue: writeback wb_workfn (flush-253:2)
Nov 23 16:22:57 HOSTNAME kernel: Call Trace:
Nov 23 16:22:57 HOSTNAME kernel: __schedule+0x2d1/0x870
Nov 23 16:22:57 HOSTNAME kernel: ? blk_flush_plug_list+0xc2/0x100
Nov 23 16:22:57 HOSTNAME kernel: schedule+0x55/0xf0
Nov 23 16:22:57 HOSTNAME kernel: raid5_get_active_stripe+0x21e/0x4e0 [raid456]
Nov 23 16:22:57 HOSTNAME kernel: ? finish_wait+0x80/0x80
Nov 23 16:22:57 HOSTNAME kernel: raid5_make_request+0x20c/0xe00 [raid456]
Nov 23 16:22:57 HOSTNAME kernel: ? do_wait_intr_irq+0xa0/0xa0
Nov 23 16:22:57 HOSTNAME kernel: ? finish_wait+0x80/0x80
Nov 23 16:22:57 HOSTNAME kernel: md_handle_request+0x128/0x1b0
Nov 23 16:22:57 HOSTNAME kernel: md_make_request+0x5b/0xb0
Nov 23 16:22:57 HOSTNAME kernel: generic_make_request_no_check+0x202/0x330
Nov 23 16:22:57 HOSTNAME kernel: submit_bio+0x3c/0x160
Nov 23 16:22:57 HOSTNAME kernel: iomap_submit_ioend.isra.42+0x4a/0x80
Nov 23 16:22:57 HOSTNAME kernel: iomap_writepage_map+0x41a/0x670
Nov 23 16:22:57 HOSTNAME kernel: write_cache_pages+0x197/0x420
Nov 23 16:22:57 HOSTNAME kernel: ? iomap_write_end+0x210/0x210
Nov 23 16:22:57 HOSTNAME kernel: iomap_writepages+0x1c/0x40
Nov 23 16:22:57 HOSTNAME kernel: xfs_vm_writepages+0x7e/0xb0 [xfs]
Nov 23 16:22:57 HOSTNAME kernel: do_writepages+0xc6/0x1d0
Nov 23 16:22:57 HOSTNAME kernel: __writeback_single_inode+0x39/0x300
Nov 23 16:22:57 HOSTNAME kernel: writeback_sb_inodes+0x1ea/0x450
Nov 23 16:22:57 HOSTNAME kernel: __writeback_inodes_wb+0x5f/0xd0
Nov 23 16:22:57 HOSTNAME kernel: wb_writeback+0x24c/0x2e0
Nov 23 16:22:57 HOSTNAME kernel: wb_workfn+0x34a/0x4d0
Nov 23 16:22:57 HOSTNAME kernel: process_one_work+0x1a7/0x360
Nov 23 16:22:57 HOSTNAME kernel: worker_thread+0x30/0x390
Nov 23 16:22:57 HOSTNAME kernel: ? create_worker+0x1a0/0x1a0
Nov 23 16:22:57 HOSTNAME kernel: kthread+0x134/0x150
Nov 23 16:22:57 HOSTNAME kernel: ? set_kthread_struct+0x50/0x50
Nov 23 16:22:57 HOSTNAME kernel: ret_from_fork+0x35/0x40
Nov 23 16:22:57 HOSTNAME kernel: INFO: task kworker/96:3:966583 blocked for more than 120 seconds.
Nov 23 16:22:57 HOSTNAME kernel:      Tainted: P           OE    --------- -  - 4.18.0-477.21.1.el8_8.x86_64 #1
Nov 23 16:22:57 HOSTNAME kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 23 16:22:57 HOSTNAME kernel: task:kworker/96:3    state:D stack:    0 pid:966583 ppid:     2 flags:0x80004080
Nov 23 16:22:57 HOSTNAME kernel: Workqueue: xfs-sync/dm-2 xfs_log_worker [xfs]
Nov 23 16:22:57 HOSTNAME kernel: Call Trace:
Nov 23 16:22:57 HOSTNAME kernel: __schedule+0x2d1/0x870
Nov 23 16:22:57 HOSTNAME kernel: schedule+0x55/0xf0
Nov 23 16:22:57 HOSTNAME kernel: schedule_timeout+0x278/0x300
Nov 23 16:22:57 HOSTNAME kernel: ? select_task_rq_fair+0x355/0x990
Nov 23 16:22:57 HOSTNAME kernel: wait_for_completion+0x96/0x100
Nov 23 16:22:57 HOSTNAME kernel: flush_work+0x11d/0x1d0
Nov 23 16:22:57 HOSTNAME kernel: ? rcu_free_pwq+0x20/0x20
Nov 23 16:22:57 HOSTNAME kernel: xlog_cil_force_seq+0x78/0x220 [xfs]
Nov 23 16:22:57 HOSTNAME kernel: xfs_log_force+0x7c/0x230 [xfs]
Nov 23 16:22:57 HOSTNAME kernel: xfs_log_worker+0x35/0x60 [xfs]
Nov 23 16:22:57 HOSTNAME kernel: process_one_work+0x1a7/0x360
Nov 23 16:22:57 HOSTNAME kernel: ? create_worker+0x1a0/0x1a0
Nov 23 16:22:57 HOSTNAME kernel: worker_thread+0x30/0x390
Nov 23 16:22:57 HOSTNAME kernel: ? create_worker+0x1a0/0x1a0
Nov 23 16:22:57 HOSTNAME kernel: kthread+0x134/0x150
Nov 23 16:22:57 HOSTNAME kernel: ? set_kthread_struct+0x50/0x50
Nov 23 16:22:57 HOSTNAME kernel: ret_from_fork+0x35/0x40
Nov 23 16:22:57 HOSTNAME kernel: INFO: task kworker/98:0:968427 blocked for more than 120 seconds.
Nov 23 16:22:57 HOSTNAME kernel:      Tainted: P           OE    --------- -  - 4.18.0-477.21.1.el8_8.x86_64 #1
Nov 23 16:22:57 HOSTNAME kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 23 16:22:57 HOSTNAME kernel: task:kworker/98:0    state:D stack:    0 pid:968427 ppid:     2 flags:0x80004080
Nov 23 16:22:57 HOSTNAME kernel: Workqueue: kdmflush/253:2 dm_wq_work [dm_mod]
Nov 23 16:22:57 HOSTNAME kernel: Call Trace:
Nov 23 16:22:57 HOSTNAME kernel: __schedule+0x2d1/0x870
Nov 23 16:22:57 HOSTNAME kernel: schedule+0x55/0xf0
Nov 23 16:22:57 HOSTNAME kernel: raid5_get_active_stripe+0x21e/0x4e0 [raid456]
Nov 23 16:22:57 HOSTNAME kernel: ? finish_wait+0x80/0x80
Nov 23 16:22:57 HOSTNAME kernel: raid5_make_request+0x20c/0xe00 [raid456]
Nov 23 16:22:57 HOSTNAME kernel: ? do_wait_intr_irq+0xa0/0xa0
Nov 23 16:22:57 HOSTNAME kernel: ? generic_make_request_checks+0x2a5/0x4f0
Nov 23 16:22:57 HOSTNAME kernel: ? dm_io_acct+0x49/0xe0 [dm_mod]
Nov 23 16:22:57 HOSTNAME kernel: ? finish_wait+0x80/0x80
Nov 23 16:22:57 HOSTNAME kernel: md_handle_request+0x128/0x1b0
Nov 23 16:22:57 HOSTNAME kernel: md_make_request+0x5b/0xb0
Nov 23 16:22:57 HOSTNAME kernel: generic_make_request_no_check+0x202/0x330
Nov 23 16:22:57 HOSTNAME kernel: dm_wq_work+0x3d/0x80 [dm_mod]
Nov 23 16:22:57 HOSTNAME kernel: process_one_work+0x1a7/0x360
Nov 23 16:22:57 HOSTNAME kernel: ? create_worker+0x1a0/0x1a0
Nov 23 16:22:57 HOSTNAME kernel: worker_thread+0x30/0x390
Nov 23 16:22:57 HOSTNAME kernel: ? create_worker+0x1a0/0x1a0
Nov 23 16:22:57 HOSTNAME kernel: kthread+0x134/0x150
Nov 23 16:22:57 HOSTNAME kernel: ? set_kthread_struct+0x50/0x50
Nov 23 16:22:57 HOSTNAME kernel: ret_from_fork+0x35/0x40
redacted-trace-kvm.txt (25,465 bytes)   
Perry MacDonald

Perry MacDonald

2023-12-01 02:56

reporter   ~0005215

Further investigations and testing - looks like my issue may be unrelated to this one as https://access.redhat.com/solutions/7029392 (raid5_get_active_stripe deadlock) seems to line up better now that I've really dug into the trace. My apologies.

Issue History

Date Modified Username Field Change
2023-10-19 09:46 Dominika Wanat New Issue
2023-10-19 09:46 Dominika Wanat Tag Attached: io hang
2023-10-19 09:46 Dominika Wanat Tag Attached: kernel
2023-10-19 09:46 Dominika Wanat Tag Attached: mdraid
2023-11-24 04:02 Perry MacDonald Note Added: 0005203
2023-11-24 04:02 Perry MacDonald File Added: redacted-trace-kvm.txt
2023-12-01 02:56 Perry MacDonald Note Added: 0005215