0001057: rngd.service sometimes fails on upgrade from Rocky 8.6 to 8.7 - Rocky Linux BugTracker

ID	Project	Category	View Status	Date Submitted	Last Update

0001057	Rocky-Linux-8	rng-tools	public	2022-11-23 09:30	2022-11-25 14:55

Reporter	Robert Sjöblom	Assigned To	Louis Abel
Priority	normal	Severity	minor	Reproducibility	have not tried
Status	needinfo	Resolution	open

Summary	0001057: rngd.service sometimes fails on upgrade from Rocky 8.6 to 8.7
Description	On dnf update from 8.6 to 8.7, there appears to be an SELinux dependency problem for rngd. This is causing the service to fail since it appears to be updated before the SELinux package. From syslog: messages:Nov 23 06:10:28 hostname dracut[58625]: * Including module: rngd * messages:Nov 23 06:10:43 hostname rngd[1084]: [rdrand]: Shutting down messages:Nov 23 06:10:43 hostname rngd[1084]: [jitter]: Shutting down messages:Nov 23 06:10:43 hostname systemd[1]: rngd.service: Succeeded. messages:Nov 23 06:10:43 hostname rngd[63797]: Disabling 7: PKCS11 Entropy generator (pkcs11) messages:Nov 23 06:10:43 hostname rngd[63797]: Disabling 5: NIST Network Entropy Beacon (nist) messages:Nov 23 06:10:43 hostname rngd[63797]: User 'daemon' not found messages:Nov 23 06:10:43 hostname systemd[1]: rngd.service: Main process exited, code=exited, status=1/FAILURE messages:Nov 23 06:10:43 hostname systemd[1]: rngd.service: Failed with result 'exit-code'. messages:Nov 23 06:10:46 hostname setroubleshoot[63801]: SELinux is preventing /usr/sbin/rngd from search access on the directory /var/lib/sss/mc/passwd. For complete SELinux messages run: sealert -l 3ca404aa-47da-4f94-959b-35d41eceaf96 messages:Nov 23 06:10:46 hostname setroubleshoot[63801]: SELinux is preventing /usr/sbin/rngd from search access on the directory /var/lib/sss/mc/passwd.#012#012*** Plugin restorecon (99.5 confidence) suggests ********************#012#012If you want to fix the label. #012/var/lib/sss/mc/passwd default label should be sssd_public_t.#012Then you can run restorecon. The access attempt may have been stopped due to insufficient permissions to access a parent directory in which case try to change the following command accordingly.#012Do#012# /sbin/restorecon -v /var/lib/sss/mc/passwd#012#012* Plugin catchall (1.49 confidence) suggests **********************#012#012If you believe that rngd should be allowed search access on the passwd directory by default.#012Then you should report this as a bug.#012You can generate a local policy module to allow this access.#012Do#012allow this access for now by executing:#012# ausearch -c 'rngd' --raw \| audit2allow -M my-rngd#012# semodule -X 300 -i my-rngd.pp#012 messages:Nov 23 06:10:46 hostname setroubleshoot[63801]: SELinux is preventing /usr/sbin/rngd from search access on the directory /var/lib/sss/mc/passwd. For complete SELinux messages run: sealert -l 3ca404aa-47da-4f94-959b-35d41eceaf96 messages:Nov 23 06:10:46 hostname setroubleshoot[63801]: SELinux is preventing /usr/sbin/rngd from search access on the directory /var/lib/sss/mc/passwd.#012#012* Plugin restorecon (99.5 confidence) suggests ********************#012#012If you want to fix the label. #012/var/lib/sss/mc/passwd default label should be sssd_public_t.#012Then you can run restorecon. The access attempt may have been stopped due to insufficient permissions to access a parent directory in which case try to change the following command accordingly.#012Do#012# /sbin/restorecon -v /var/lib/sss/mc/passwd#012#012* Plugin catchall (1.49 confidence) suggests **********************#012#012If you believe that rngd should be allowed search access on the passwd directory by default.#012Then you should report this as a bug.#012You can generate a local policy module to allow this access.#012Do#012allow this access for now by executing:#012# ausearch -c 'rngd' --raw \| audit2allow -M my-rngd#012# semodule -X 300 -i my-rngd.pp#012 messages:Nov 23 06:10:47 hostname setroubleshoot[63801]: SELinux is preventing /usr/sbin/rngd from search access on the directory /var/lib/sss/pipes/nss. For complete SELinux messages run: sealert -l 3ca404aa-47da-4f94-959b-35d41eceaf96 messages:Nov 23 06:10:47 hostname setroubleshoot[63801]: SELinux is preventing /usr/sbin/rngd from search access on the directory /var/lib/sss/pipes/nss.#012#012* Plugin restorecon (99.5 confidence) suggests ********************#012#012If you want to fix the label. #012/var/lib/sss/pipes/nss default label should be sssd_public_t.#012Then you can run restorecon. The access attempt may have been stopped due to insufficient permissions to access a parent directory in which case try to change the following command accordingly.#012Do#012# /sbin/restorecon -v /var/lib/sss/pipes/nss#012#012* Plugin catchall (1.49 confidence) suggests **********************#012#012If you believe that rngd should be allowed search access on the nss directory by default.#012Then you should report this as a bug.#012You can generate a local policy module to allow this access.#012Do#012allow this access for now by executing:#012# ausearch -c 'rngd' --raw \| audit2allow -M my-rngd#012# semodule -X 300 -i my-rngd.pp#012 messages:Nov 23 06:10:47 hostname setroubleshoot[63801]: SELinux is preventing /usr/sbin/rngd from read access on the file /etc/passwd. For complete SELinux messages run: sealert -l 2ab94b3a-95dc-466c-a32d-5162e4b6a8f3 messages:Nov 23 06:10:47 hostname setroubleshoot[63801]: SELinux is preventing /usr/sbin/rngd from read access on the file /etc/passwd.#012#012* Plugin catchall (100. confidence) suggests **********************#012#012If you believe that rngd should be allowed read access on the passwd file by default.#012Then you should report this as a bug.#012You can generate a local policy module to allow this access.#012Do#012allow this access for now by executing:#012# ausearch -c 'rngd' --raw \| audit2allow -M my-rngd#012# semodule -X 300 -i my-rngd.pp#012 messages:Nov 23 06:10:47 hostname setroubleshoot[63801]: SELinux is preventing /usr/sbin/rngd from search access on the directory /run/dbus/system_bus_socket. For complete SELinux messages run: sealert -l d16511fa-16aa-4e70-a0a9-8125f4b8d336 messages:Nov 23 06:10:47 hostname setroubleshoot[63801]: SELinux is preventing /usr/sbin/rngd from search access on the directory /run/dbus/system_bus_socket.#012#012* Plugin catchall (100. confidence) suggests ************************#012#012If you believe that rngd should be allowed search access on the system_bus_socket directory by default.#012Then you should report this as a bug.#012You can generate a local policy module to allow this access.#012Do#012allow this access for now by executing:#012# ausearch -c 'rngd' --raw \| audit2allow -M my-rngd#012# semodule -X 300 -i my-rngd.pp#012 From dnf.rpm.log: 2022-11-23T06:09:56+0100 SUBDEBUG Upgrade: rng-tools-6.15-1.el8.x86_64 2022-11-23T06:10:43+0100 SUBDEBUG Upgraded: rng-tools-6.14-6.git.b2b7934e.el8_6.x86_64 ... 2022-11-23T06:12:47+0100 SUBDEBUG Upgraded: ipa-selinux-4.9.8-8.module+el8.6.0+1050+4989852e.noarch 2022-11-23T06:12:47+0100 SUBDEBUG Upgraded: rpm-plugin-selinux-4.14.3-24.el8_6.x86_64 2022-11-23T06:12:47+0100 SUBDEBUG Upgraded: selinux-policy-targeted-3.14.3-95.el8_6.4.noarch 2022-11-23T06:12:48+0100 SUBDEBUG Upgraded: selinux-policy-3.14.3-95.el8_6.4.noarch 2022-11-23T06:12:50+0100 SUBDEBUG Upgraded: python3-libselinux-2.9-5.el8.x86_64 2022-11-23T06:12:50+0100 SUBDEBUG Upgraded: libselinux-utils-2.9-5.el8.x86_64 2022-11-23T06:13:01+0100 SUBDEBUG Upgraded: libselinux-2.9-5.el8.x86_64 Here we can see rng-tools are upgraded before SELinux package.
Tags	No tags attached.

Robert Sjöblom 2022-11-23 09:30 reporter ~0001222	Restarting rngd after package SELinux has been upgraded succeeds

Robert Sjöblom 2022-11-24 08:52 reporter ~0001255	In some cases, restarting rngd fails with "can't find user daemon"; rngd is prevented from reading the file by selinux. Solution is to reinstall selinux-policy package, then restart rngd.

Louis Abel 2022-11-24 09:39 administrator ~0001256	I am unable to replicate this issue. After patching and rebooting an 8.6 system, rngd starts up as expected and there are no selinux errors. Reinstalling the selinux-policy package leads me to believe there's either a configuration issue or there is possibly an edge case you've ran into. Can you provide your entire dnf.rpm.log of the day you ran dnf update? [root@router scsi]# uname -r 4.18.0-372.26.1.el8_6.x86_64 [root@router scsi]# dnf update -y -q [root@router scsi]# rpm -q rng-tools rng-tools-6.15-1.el8.x86_64 [root@router scsi]# rpm -q selinux-policy selinux-policy-3.14.3-108.el8.noarch [root@router scsi]# systemctl status rngd ● rngd.service - Hardware RNG Entropy Gatherer Daemon Loaded: loaded (/usr/lib/systemd/system/rngd.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2022-11-15 11:06:42 MST; 1 weeks 1 days ago Main PID: 333967 (rngd) Tasks: 5 (limit: 409620) Memory: 1.9M CGroup: /system.slice/rngd.service └─333967 /usr/sbin/rngd -f --fill-watermark=0 -x pkcs11 -x nist -D daemon:daemon Nov 15 11:06:42 router systemd[1]: Started Hardware RNG Entropy Gatherer Daemon. Nov 15 11:06:42 router rngd[333967]: Disabling 7: PKCS11 Entropy generator (pkcs11) Nov 15 11:06:42 router rngd[333967]: Disabling 5: NIST Network Entropy Beacon (nist) Nov 15 11:06:42 router rngd[333967]: Initializing available sources Nov 15 11:06:42 router rngd[333967]: [hwrng ]: Initialized Nov 15 11:06:42 router rngd[333967]: [rdrand]: Initialization Failed Nov 15 11:06:42 router rngd[333967]: [jitter]: Initializing AES buffer Nov 15 11:06:47 router rngd[333967]: [jitter]: Enabling JITTER rng support Nov 15 11:06:47 router rngd[333967]: [jitter]: Initialized Nov 15 11:06:47 router rngd[333967]: Process privileges have been dropped to 2:2 [root@router scsi]# init 6 [root@router ~]# systemctl status rngd ● rngd.service - Hardware RNG Entropy Gatherer Daemon Loaded: loaded (/usr/lib/systemd/system/rngd.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2022-11-24 02:15:32 MST; 1min 15s ago Main PID: 1537 (rngd) Tasks: 5 (limit: 409712) Memory: 6.5M CGroup: /system.slice/rngd.service └─1537 /usr/sbin/rngd -f --fill-watermark=0 -x pkcs11 -x nist -D daemon:daemon Nov 24 02:15:32 router systemd[1]: Started Hardware RNG Entropy Gatherer Daemon. Nov 24 02:15:32 router rngd[1537]: Disabling 7: PKCS11 Entropy generator (pkcs11) Nov 24 02:15:32 router rngd[1537]: Disabling 5: NIST Network Entropy Beacon (nist) Nov 24 02:15:32 router rngd[1537]: Initializing available sources Nov 24 02:15:32 router rngd[1537]: [hwrng ]: Initialized Nov 24 02:15:32 router rngd[1537]: [rdrand]: Initialization Failed Nov 24 02:15:32 router rngd[1537]: [jitter]: Initializing AES buffer Nov 24 02:15:37 router rngd[1537]: [jitter]: Enabling JITTER rng support Nov 24 02:15:37 router rngd[1537]: [jitter]: Initialized Nov 24 02:15:37 router rngd[1537]: Process privileges have been dropped to 2:2 [root@router ~]# audit2why < /var/log/audit/audit.log [root@router ~]# [root@router ~]# grep -En 'rng\|selinux' /tmp/dnf.rpm.log 5:2022-11-24T01:56:03-0700 SUBDEBUG Upgrade: libselinux-2.9-6.el8.x86_64 44:2022-11-24T01:56:08-0700 SUBDEBUG Upgrade: libselinux-utils-2.9-6.el8.x86_64 95:2022-11-24T01:56:19-0700 SUBDEBUG Upgrade: libselinux-devel-2.9-6.el8.x86_64 144:2022-11-24T01:57:05-0700 SUBDEBUG Upgrade: python3-libselinux-2.9-6.el8.x86_64 196:2022-11-24T01:57:21-0700 SUBDEBUG Upgrade: rpm-plugin-selinux-4.14.3-24.el8_7.x86_64 197:2022-11-24T01:57:21-0700 SUBDEBUG Upgrade: selinux-policy-3.14.3-108.el8.noarch 198:2022-11-24T01:57:39-0700 SUBDEBUG Upgrade: selinux-policy-targeted-3.14.3-108.el8.noarch 247:2022-11-24T01:58:38-0700 SUBDEBUG Upgrade: ipa-selinux-4.9.10-3.module+el8.7.0+1074+aae18f3a.noarch 398:2022-11-24T02:05:30-0700 SUBDEBUG Upgrade: rng-tools-6.15-1.el8.x86_64 522:2022-11-24T02:06:20-0700 SUBDEBUG Upgraded: libselinux-devel-2.9-5.el8.x86_64 540:2022-11-24T02:06:22-0700 SUBDEBUG Upgraded: ipa-selinux-4.9.8-6.module+el8.6.0+797+07647629.noarch 582:2022-11-24T02:06:42-0700 SUBDEBUG Upgraded: rng-tools-6.14-6.git.b2b7934e.el8_6.x86_64 814:2022-11-24T02:08:36-0700 SUBDEBUG Upgraded: rpm-plugin-selinux-4.14.3-24.el8_6.x86_64 815:2022-11-24T02:08:36-0700 SUBDEBUG Upgraded: selinux-policy-targeted-3.14.3-95.el8_6.4.noarch 816:2022-11-24T02:08:37-0700 SUBDEBUG Upgraded: selinux-policy-3.14.3-95.el8_6.4.noarch 842:2022-11-24T02:08:39-0700 SUBDEBUG Upgraded: python3-libselinux-2.9-5.el8.x86_64 848:2022-11-24T02:08:39-0700 SUBDEBUG Upgraded: libselinux-utils-2.9-5.el8.x86_64 910:2022-11-24T02:08:56-0700 SUBDEBUG Upgraded: libselinux-2.9-5.el8.x86_64

Robert Sjöblom 2022-11-25 14:55 reporter ~0001288	We've only seen this issue on a few of the servers in the fleet, namely the postgres servers. We're currently running ~900 postgres servers, out of ~1600 machines total. Around 10 of them have been affected by this issue, so it seems likely that it's an edge condition of some kind. We use ansible to configure all our servers, and so they should be exactly similar in configuration. The servers we have seen this issue on were initially installed with CentOS 8, and later converted to Rocky using Rocky's conversion script. It's possible that it's related, but it's also a fact that the large majority of servers are in this state. We have much fewer new servers with a clean Rocky 8 install. When looking into the dnf rpm log during the upgrade window, we found an SELinux policy error due to a memory allocation failure. 2022-11-24T05:43:00+0100 SUBDEBUG Upgrade: selinux-policy-targeted-3.14.3-108.el8.noarch 2022-11-24T05:43:38+0100 INFO SELinux: Could not load policy file /etc/selinux/targeted/policy/policy.31: Cannot allocate memory 2022-11-24T05:43:38+0100 SUBDEBUG Upgrade: sssd-dbus-2.7.3-4.el8_7.1.x86_64 The system journal at the same time shows the following: Nov 24 05:43:37 hostname kernel: load_policy: page allocation failure: order:4, mode:0x60c0c0(GFP_KERNEL\|__GFP_COMP\|__GFP_ZERO), nodemask=(null),cpuset=/,mems_allowed=0 Nov 24 05:43:37 hostname kernel: CPU: 1 PID: 2351056 Comm: load_policy Kdump: loaded Not tainted 4.18.0-372.9.1.el8.x86_64 #1 Nov 24 05:43:37 hostname kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020 Nov 24 05:43:37 hostname kernel: Call Trace: Nov 24 05:43:37 hostname kernel: dump_stack+0x41/0x60 Nov 24 05:43:37 hostname kernel: warn_alloc.cold.119+0x7b/0x111 Nov 24 05:43:37 hostname kernel: ? _cond_resched+0x15/0x30 Nov 24 05:43:37 hostname kernel: ? __alloc_pages_direct_compact+0x157/0x160 Nov 24 05:43:37 hostname kernel: __alloc_pages_slowpath+0xc7e/0xcc0 Nov 24 05:43:37 hostname kernel: ? type_read+0x160/0x160 Nov 24 05:43:37 hostname kernel: __alloc_pages_nodemask+0x2db/0x310 Nov 24 05:43:37 hostname kernel: kmalloc_order+0x28/0x90 Nov 24 05:43:37 hostname kernel: kmalloc_order_trace+0x1d/0xa0 Nov 24 05:43:37 hostname kernel: ? type_read+0x160/0x160 Nov 24 05:43:37 hostname kernel: __kmalloc+0x1ff/0x250 Nov 24 05:43:37 hostname kernel: ? type_read+0x160/0x160 Nov 24 05:43:37 hostname kernel: hashtab_init+0x5d/0x80 Nov 24 05:43:37 hostname kernel: policydb_read+0x2e3/0x1230 Nov 24 05:43:37 hostname kernel: security_load_policy+0xa8/0x5e0 Nov 24 05:43:37 hostname kernel: ? copy_user_generic_unrolled+0x32/0xc0 Nov 24 05:43:37 hostname kernel: sel_write_load+0xde/0x1a0 Nov 24 05:43:37 hostname kernel: vfs_write+0xa5/0x1a0 Nov 24 05:43:37 hostname kernel: ksys_write+0x4f/0xb0 Nov 24 05:43:37 hostname kernel: do_syscall_64+0x5b/0x1a0 Nov 24 05:43:37 hostname kernel: entry_SYSCALL_64_after_hwframe+0x65/0xca Nov 24 05:43:37 hostname kernel: RIP: 0033:0x7fa9bd1f2bc8 Nov 24 05:43:37 hostname kernel: Code: 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 55 4b 2a 00 8b 00 85 c0 75 17 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 58 c3 0f 1f 80 00 00 00 00 41 54 49 89 d4 55 Nov 24 05:43:37 hostname kernel: RSP: 002b:00007ffe2428f228 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 Nov 24 05:43:37 hostname kernel: RAX: ffffffffffffffda RBX: 00007ffe2428f230 RCX: 00007fa9bd1f2bc8 Nov 24 05:43:37 hostname kernel: RDX: 0000000000833419 RSI: 00007fa9af240000 RDI: 0000000000000004 Nov 24 05:43:37 hostname kernel: RBP: 0000000000000004 R08: 000055cc2d2e22a0 R09: 00007fa9bd252d40 Nov 24 05:43:37 hostname kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 00007fa9af240000 Nov 24 05:43:37 hostname kernel: R13: 0000000000833419 R14: 000000000000000f R15: 0000000000000003 Nov 24 05:43:37 hostname kernel: Mem-Info: Nov 24 05:43:37 hostname kernel: active_anon:17640 inactive_anon:182469 isolated_anon:0 active_file:812856 inactive_file:1292224 isolated_file:0 unevictable:0 dirty:6 writeback:0 slab_reclaimable:170338 slab_unreclaimable:66719 mapped:57643 shmem:41017 pagetables:4920 bounce:0 free:135449 free_pcp:0 free_cma:0 Nov 24 05:43:37 hostname kernel: Node 0 active_anon:70560kB inactive_anon:729876kB active_file:3251424kB inactive_file:5168896kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:230572kB dirty:24kB writeback:0kB shmem:164068kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 12288kB writeback_tmp:0kB kernel_stack:8400kB pagetables:19680kB all_unreclaimable? no Nov 24 05:43:37 hostname kernel: Node 0 DMA free:13312kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB Nov 24 05:43:37 hostname kernel: lowmem_reserve[]: 0 2768 15745 15745 15745 Nov 24 05:43:37 hostname kernel: Node 0 DMA32 free:434312kB min:11972kB low:14964kB high:17956kB active_anon:7344kB inactive_anon:347200kB active_file:655228kB inactive_file:936252kB unevictable:0kB writepending:16kB present:3129152kB managed:2867008kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB Nov 24 05:43:37 hostname kernel: lowmem_reserve[]: 0 0 12977 12977 12977 Nov 24 05:43:37 hostname kernel: Node 0 Normal free:94172kB min:55544kB low:69428kB high:83312kB active_anon:63216kB inactive_anon:382852kB active_file:2596196kB inactive_file:4232836kB unevictable:0kB writepending:8kB present:13631488kB managed:13297412kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB Nov 24 05:43:37 hostname kernel: lowmem_reserve[]: 0 0 0 0 0 Nov 24 05:43:37 hostname kernel: Node 0 DMA: 04kB 08kB 016kB 032kB 064kB 0128kB 0256kB 0512kB 11024kB (U) 22048kB (UM) 24096kB (M) = 13312kB Nov 24 05:43:37 hostname kernel: Node 0 DMA32: 240384kB (UME) 220938kB (UME) 955116kB (UME) 29332kB (UME) 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 04096kB = 435088kB Nov 24 05:43:37 hostname kernel: Node 0 Normal: 82674kB (UMEH) 11928kB (UMEH) 240316kB (UMEH) 39132kB (UH) 1064kB (H) 3128kB (H) 0256kB 0512kB 01024kB 02048kB 04096kB = 94588kB Nov 24 05:43:37 hostname kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Nov 24 05:43:37 hostname kernel: Node 0 hugepages_total=2400 hugepages_free=287 hugepages_surp=0 hugepages_size=2048kB Nov 24 05:43:37 hostname kernel: 2140592 total pagecache pages Nov 24 05:43:37 hostname kernel: 65 pages in swap cache Nov 24 05:43:37 hostname kernel: Swap cache stats: add 217664, delete 217599, find 49323/56024 Nov 24 05:43:37 hostname kernel: Free swap = 1385212kB Nov 24 05:43:37 hostname kernel: Total swap = 2064380kB Nov 24 05:43:37 hostname kernel: 4194157 pages RAM Nov 24 05:43:37 hostname kernel: 0 pages HighMem/MovableOnly Nov 24 05:43:37 hostname kernel: 149212 pages reserved Nov 24 05:43:37 hostname kernel: 0 pages hwpoisoned Nov 24 05:43:37 hostname kernel: SELinux: failed to load policy Nov 24 05:43:38 hostname dnf-automatic[2350061]: SELinux: Could not load policy file /etc/selinux/targeted/policy/policy.31: Cannot allocate memory Nov 24 05:43:38 hostname dnf-automatic[2350061]: load_policy: Can't load policy: Cannot allocate memory Nov 24 05:43:38 hostname dbus-daemon[1100]: [system] Reloaded configuration Nov 24 05:43:38 hostname dbus-daemon[1100]: [system] Reloaded configuration Nov 24 05:43:38 hostname dbus-daemon[1100]: [system] Reloaded configuration Nov 24 05:43:38 hostname dbus-daemon[1100]: [system] Reloaded configuration Nov 24 05:43:40 hostname dbus-daemon[1100]: [system] Reloaded configuration Nov 24 05:43:40 hostname dbus-daemon[1100]: [system] Reloaded configuration Nov 24 05:43:41 hostname systemd-udevd[2351902]: Using default interface naming scheme 'rhel-8.0'. Nov 24 05:43:42 hostname systemd[1]: Reloading. Nov 24 05:43:43 hostname systemd[1]: Reloading. Nov 24 05:43:47 hostname NetworkManager[1238]: <info> [1669265027.8825] manager: kernel firmware directory '/lib/firmware' changed Due to this server being a database host, we have allocated hugepages and disabled overcommit. We have 2400 2048 kB memory allocated for hugepages and 11076044 kB available for userspace applications (CommitLimit). Total memory on the machine is, according to /proc/meminfo, 16179780 kB which should leave 188536 kB for the kernel. Perhaps this is not enough? Here's our cat /proc/meminfo for one of the affected servers: MemTotal: 16179780 kB MemFree: 384992 kB MemAvailable: 9381080 kB Buffers: 3704 kB Cached: 8809540 kB SwapCached: 276 kB Active: 3656528 kB Inactive: 5692040 kB Active(anon): 83920 kB Inactive(anon): 611336 kB Active(file): 3572608 kB Inactive(file): 5080704 kB Unevictable: 0 kB Mlocked: 0 kB SwapTotal: 2064380 kB SwapFree: 1387004 kB Dirty: 92 kB Writeback: 0 kB AnonPages: 531136 kB Mapped: 257248 kB Shmem: 189552 kB KReclaimable: 686400 kB Slab: 953032 kB SReclaimable: 686400 kB SUnreclaim: 266632 kB KernelStack: 8384 kB PageTables: 19500 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 11076044 kB Committed_AS: 2940032 kB VmallocTotal: 34359738367 kB VmallocUsed: 0 kB VmallocChunk: 0 kB Percpu: 101888 kB HardwareCorrupted: 0 kB AnonHugePages: 10240 kB ShmemHugePages: 0 kB ShmemPmdMapped: 0 kB FileHugePages: 0 kB FilePmdMapped: 0 kB HugePages_Total: 2400 HugePages_Free: 287 HugePages_Rsvd: 9 HugePages_Surp: 0 Hugepagesize: 2048 kB Hugetlb: 4915200 kB DirectMap4k: 1275712 kB DirectMap2M: 12355584 kB DirectMap1G: 5242880 kB

Date Modified	Username	Field	Change
2022-11-23 09:30	Robert Sjöblom	New Issue
2022-11-23 09:30	Robert Sjöblom	Note Added: 0001222
2022-11-24 08:52	Robert Sjöblom	Note Added: 0001255
2022-11-24 09:39	Louis Abel	Assigned To	=> Louis Abel
2022-11-24 09:39	Louis Abel	Status	new => needinfo
2022-11-24 09:39	Louis Abel	Note Added: 0001256
2022-11-25 14:55	Robert Sjöblom	Note Added: 0001288