View Issue Details
ID | Project | Category | View Status | Date Submitted | Last Update |
---|---|---|---|---|---|
0000179 | Rocky-Linux-9 | openssh | public | 2022-08-11 21:57 | 2022-08-12 19:55 |
Reporter | Roger Simmons | Assigned To | |||
Priority | normal | Severity | major | Reproducibility | always |
Status | closed | Resolution | no change required | ||
Summary | 0000179: ssh sessions do not inherit taskset of parent sshd process | ||||
Description | init (pid 1) is taskset to a subset of cores. sshd is started and inherits the taskset from pid 1 as expected. Logging onto the system via ssh, the resulting sshd child and grandchild taskset is "all cores" rather than inheriting the parent sshd process taskset Appears to be specific to Rocky 9 (not seen in 8.5) (Tagged as major as this breaks standard taskset parent->child inheritance which can be relied on more widely) | ||||
Steps To Reproduce | 1. taskset pid 1 to a subset of the online cpus 2. restart sshd 3. confirm sshd process has inherited the correct taskset from pid 1 (taskset -p <sshd pid>) 4. ssh onto the system; raises 2 new sshd child processes, the first <sshd_pid> as parent, the second with <sshd_pid> as grandparent 5. taskset -p the 2 new sshd child processes from step 4 shows a different taskset to pid 1 and <sshd_pid>, but should be inherited | ||||
Additional Information | $ ps -aef | grep sshd root 1515 1 0 11:14 ? 00:00:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups root 266390 1515 0 22:20 ? 00:00:00 sshd: user [priv] user 266429 266390 0 22:20 ? 00:00:00 sshd: user@pts/0 $ taskset -p 1 pid 1's current affinity mask: fe03 <------ correct: intentional subset of all cores $ taskset -p 1515 pid 1515's current affinity mask: fe03 <----- correct: inherits pid 1 as expected $ taskset -p 266390 pid 266390's current affinity mask: ffff <----- should be fe03 (inherited from pid 1515) $ taskset -p 266444 pid 266444's current affinity mask: ffff <----- should be fe03 (once pid 266390 corrected) | ||||
Tags | sshd, taskset | ||||
Hello. Thank you for the report. I cannot reproduce this behavior under normal circumstances. [label@awx ~]$ ps -aef | grep sshd root 896 1 0 Aug03 ? 00:00:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups root 1525 896 0 Aug03 ? 00:00:00 sshd: label [priv] label 1540 1525 0 Aug03 ? 00:00:00 sshd: label@pts/0 root 10144 896 0 15:22 ? 00:00:00 sshd: label [priv] label 10147 10144 0 15:22 ? 00:00:00 sshd: label@pts/2 label 10174 10148 0 15:22 pts/2 00:00:00 grep --color=auto sshd [label@awx ~]$ taskset -p 1 pid 1's current affinity mask: f [label@awx ~]$ taskset -p 896 pid 896's current affinity mask: f [label@awx ~]$ taskset -p 1525 pid 1525's current affinity mask: f [label@awx ~]$ taskset -p 1540 pid 1540's current affinity mask: f [label@awx ~]$ taskset -p 10144 pid 10144's current affinity mask: f [label@awx ~]$ taskset -p 10147 pid 10147's current affinity mask: f When restarting sshd, I still cannot reproduce it. [label@awx ~]$ sudo systemctl restart sshd [sudo] password for label: [label@awx ~]$ ps -aef | grep sshd root 1525 1 0 Aug03 ? 00:00:00 sshd: label [priv] label 1540 1525 0 Aug03 ? 00:00:00 sshd: label@pts/0 root 10144 1 0 15:22 ? 00:00:00 sshd: label [priv] label 10147 10144 0 15:22 ? 00:00:00 sshd: label@pts/2 root 10194 1 0 15:25 ? 00:00:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups label 10196 10148 0 15:25 pts/2 00:00:00 grep --color=auto sshd [label@awx ~]$ taskset -p 1 pid 1's current affinity mask: f [label@awx ~]$ taskset -p 10194 pid 10194's current affinity mask: f [label@awx ~]$ taskset -p 1525 pid 1525's current affinity mask: f [label@awx ~]$ taskset -p 1540 pid 1540's current affinity mask: f [label@awx ~]$ taskset -p 10144 pid 10144's current affinity mask: f [label@awx ~]$ taskset -p 10147 pid 10147's current affinity mask: f Setting to need info. |
|
Thanks Louis - can I double check if you changed the taskset for pid 1/init from the default "all cores" as the first step in testing. The issue only shows when pid 1/init is set to only use a subset of cores. If in step 1 you taskset init to a subset, - eg: #> taskset 3 -p 1 - then restart sshd - and ssh back onto the machine, the child & grandchild pids will show as "all cores"/f rather then the 3 of init & sshd |
|
I did not modify the affinity before, that was a step I missed. See below. The tests were performed on Rocky Linux 9, Rocky Linux 8, CentOS Stream 9 Rocky Linux 9 # Original affinity. Current pts/2 session. [root@awx ~]# taskset -p 1 pid 1's current affinity mask: f [root@awx ~]# ps -aef | grep sshd root 1525 1 0 Aug03 ? 00:00:00 sshd: label [priv] label 1540 1525 0 Aug03 ? 00:00:00 sshd: label@pts/0 root 10194 1 0 Aug11 ? 00:00:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups root 10694 10194 0 00:26 ? 00:00:00 sshd: label [priv] label 10702 10694 0 00:26 ? 00:00:00 sshd: label@pts/2 root 10776 10728 0 00:41 pts/3 00:00:00 grep --color=auto sshd [root@awx ~]# taskset -p 10194 pid 10194's current affinity mask: f [root@awx ~]# taskset -p 10694 pid 10694's current affinity mask: f [root@awx ~]# taskset -p 10702 pid 10702's current affinity mask: f # Setting mask to 3 [root@awx ~]# taskset -p 3 1 pid 1's current affinity mask: f pid 1's new affinity mask: 3 [root@awx ~]# taskset -p 1 pid 1's current affinity mask: 3 [root@awx ~]# taskset -c -p 1 pid 1's current affinity list: 0,1 # Restarting sshd [root@awx ~]# systemctl restart sshd # checking affinity has changed to 3 for new pts/2 session, while older pts/0 session is still f [root@awx ~]# ps -aef | grep sshd root 1525 1 0 Aug03 ? 00:00:00 sshd: label [priv] label 1540 1525 0 Aug03 ? 00:00:00 sshd: label@pts/0 root 10786 1 0 00:43 ? 00:00:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups root 10788 10786 0 00:43 ? 00:00:00 sshd: label [priv] label 10797 10788 0 00:43 ? 00:00:00 sshd: label@pts/2 root 10849 10824 0 00:44 pts/3 00:00:00 grep --color=auto sshd [root@awx ~]# taskset -p 10786 pid 10786's current affinity mask: 3 [root@awx ~]# taskset -p 10788 pid 10788's current affinity mask: 3 [root@awx ~]# taskset -p 10797 pid 10797's current affinity mask: 3 --- CentOS Stream 9 # Current affinity [root@localhost ~]# taskset -p 1 pid 1's current affinity mask: f # Setting affinity to 3 [root@localhost ~]# taskset -p 3 1 pid 1's current affinity mask: f pid 1's new affinity mask: 3 [root@localhost ~]# taskset -c -p 1 pid 1's current affinity list: 0,1 [root@localhost ~]# ps -aef | grep sshd root 773 1 0 00:27 ? 00:00:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups root 1236 773 0 00:27 ? 00:00:00 sshd: root [priv] root 1251 1236 0 00:27 ? 00:00:00 sshd: root@pts/0 root 1308 1252 0 00:31 pts/0 00:00:00 grep --color=auto sshd [root@localhost ~]# taskset -p 1 pid 1's current affinity mask: 3 # Current sshd processes inherit the original mask of f [root@localhost ~]# taskset -p 773 pid 773's current affinity mask: f [root@localhost ~]# taskset -p 1236 pid 1236's current affinity mask: f [root@localhost ~]# taskset -p 1251 pid 1251's current affinity mask: f # The new sshd listener inherits 3 [root@localhost ~]# systemctl restart sshd [root@localhost ~]# ps -aef | grep sshd root 1236 1 0 00:27 ? 00:00:00 sshd: root [priv] root 1251 1236 0 00:27 ? 00:00:00 sshd: root@pts/0 root 1316 1 0 00:32 ? 00:00:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups root 1319 1252 0 00:32 pts/0 00:00:00 grep --color=auto sshd [root@localhost ~]# taskset -p 1316 pid 1316's current affinity mask: 3 # New sshd processes inherit 3 [root@localhost ~]# ps -aef | grep sshd root 1236 1 0 00:27 ? 00:00:00 sshd: root [priv] root 1251 1236 0 00:27 ? 00:00:00 sshd: root@pts/0 root 1316 1 0 00:32 ? 00:00:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups root 1324 1316 0 00:33 ? 00:00:00 sshd: root [priv] root 1328 1324 0 00:33 ? 00:00:00 sshd: root@pts/1 root 1353 1329 0 00:33 pts/1 00:00:00 grep --color=auto sshd [root@localhost ~]# taskset -p 1 pid 1's current affinity mask: 3 [root@localhost ~]# taskset -p 1316 pid 1316's current affinity mask: 3 [root@localhost ~]# taskset -p 1324 pid 1324's current affinity mask: 3 [root@localhost ~]# taskset -p 1328 pid 1328's current affinity mask: 3 --- Rocky Linux 8 [root@cm01 ~]# ps -aef | grep sshd root 1007 1 0 Aug02 ? 00:00:00 /usr/sbin/sshd -D root 4272 1007 0 Aug02 ? 00:00:00 sshd: label [priv] label 4805 4272 0 Aug02 ? 00:00:00 sshd: label@pts/0 root 17519 1007 0 00:46 ? 00:00:00 sshd: label [priv] label 17530 17519 0 00:46 ? 00:00:00 sshd: label@pts/2 root 17634 17605 0 00:46 pts/3 00:00:00 grep --color=auto sshd [root@cm01 ~]# taskset -p 1 pid 1's current affinity mask: 3 [root@cm01 ~]# taskset -p 4272 pid 4272's current affinity mask: 3 [root@cm01 ~]# taskset -p 4805 pid 4805's current affinity mask: 3 [root@cm01 ~]# taskset -p 1 1 pid 1's current affinity mask: 3 pid 1's new affinity mask: 1 [root@cm01 ~]# systemctl restart sshd # New session after restart. Affinity mask of 1 matches [root@cm01 ~]# ps -aef | grep sshd root 4272 1 0 Aug02 ? 00:00:00 sshd: label [priv] label 4805 4272 0 Aug02 ? 00:00:00 sshd: label@pts/0 root 17648 1 0 00:48 ? 00:00:00 /usr/sbin/sshd -D root 17662 17648 0 00:48 ? 00:00:00 sshd: label [priv] label 17668 17662 0 00:48 ? 00:00:00 sshd: label@pts/2 root 17749 17717 0 00:49 pts/3 00:00:00 grep --color=auto sshd [root@cm01 ~]# taskset -p 1 pid 1's current affinity mask: 1 [root@cm01 ~]# taskset -p 17648 pid 17648's current affinity mask: 1 [root@cm01 ~]# taskset -p 17662 pid 17662's current affinity mask: 1 [root@cm01 ~]# taskset -p 17668 pid 17668's current affinity mask: 1 |
|
Thank you - that all looks completely normal. Taking a further look at things this side I have tracked down the issue to an unexpected/unwanted cgroup setup. Specifically /sys/fs/cgroup/cgroup.subtree_control contains cpuset, and this was interfering with the usual init/parent/child inheritance. Removing this returns the normal/expected behaviour I have checked with a vanilla VM install of Rocky 9 and see that the default cgroup.subtree_control omits cpuset (and cpu) which is fine, so this must have been imposed by a 3rd party install somewhere along the way. Thanks again. This can be closed; no issue. |
|
Date Modified | Username | Field | Change |
---|---|---|---|
2022-08-11 21:57 | Roger Simmons | New Issue | |
2022-08-11 21:57 | Roger Simmons | Tag Attached: sshd | |
2022-08-11 21:57 | Roger Simmons | Tag Attached: taskset | |
2022-08-11 22:29 | Louis Abel | Assigned To | => Louis Abel |
2022-08-11 22:29 | Louis Abel | Status | new => needinfo |
2022-08-11 22:29 | Louis Abel | Note Added: 0000337 | |
2022-08-12 07:24 | Roger Simmons | Note Added: 0000338 | |
2022-08-12 08:03 | Louis Abel | Note Added: 0000339 | |
2022-08-12 11:23 | Roger Simmons | Note Added: 0000341 | |
2022-08-12 19:55 | Louis Abel | Assigned To | Louis Abel => |
2022-08-12 19:55 | Louis Abel | Status | needinfo => closed |
2022-08-12 19:55 | Louis Abel | Resolution | open => no change required |