View Issue Details

IDProjectCategoryView StatusLast Update
0000179Rocky-Linux-9opensshpublic2022-08-12 19:55
ReporterRoger Simmons Assigned To 
PrioritynormalSeveritymajorReproducibilityalways
Status closedResolutionno change required 
Summary0000179: ssh sessions do not inherit taskset of parent sshd process
Descriptioninit (pid 1) is taskset to a subset of cores.
sshd is started and inherits the taskset from pid 1 as expected.
Logging onto the system via ssh, the resulting sshd child and grandchild taskset is "all cores" rather than inheriting the parent sshd process taskset

Appears to be specific to Rocky 9 (not seen in 8.5)

(Tagged as major as this breaks standard taskset parent->child inheritance which can be relied on more widely)
Steps To Reproduce1. taskset pid 1 to a subset of the online cpus
2. restart sshd
3. confirm sshd process has inherited the correct taskset from pid 1 (taskset -p <sshd pid>)
4. ssh onto the system; raises 2 new sshd child processes, the first <sshd_pid> as parent, the second with <sshd_pid> as grandparent
5. taskset -p the 2 new sshd child processes from step 4 shows a different taskset to pid 1 and <sshd_pid>, but should be inherited
Additional Information$ ps -aef | grep sshd
root 1515 1 0 11:14 ? 00:00:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
root 266390 1515 0 22:20 ? 00:00:00 sshd: user [priv]
user 266429 266390 0 22:20 ? 00:00:00 sshd: user@pts/0

$ taskset -p 1
pid 1's current affinity mask: fe03 <------ correct: intentional subset of all cores

$ taskset -p 1515
pid 1515's current affinity mask: fe03 <----- correct: inherits pid 1 as expected

$ taskset -p 266390
pid 266390's current affinity mask: ffff <----- should be fe03 (inherited from pid 1515)

$ taskset -p 266444
pid 266444's current affinity mask: ffff <----- should be fe03 (once pid 266390 corrected)
Tagssshd, taskset

Activities

Louis Abel

Louis Abel

2022-08-11 22:29

administrator   ~0000337

Hello. Thank you for the report. I cannot reproduce this behavior under normal circumstances.

[label@awx ~]$ ps -aef | grep sshd
root 896 1 0 Aug03 ? 00:00:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
root 1525 896 0 Aug03 ? 00:00:00 sshd: label [priv]
label 1540 1525 0 Aug03 ? 00:00:00 sshd: label@pts/0
root 10144 896 0 15:22 ? 00:00:00 sshd: label [priv]
label 10147 10144 0 15:22 ? 00:00:00 sshd: label@pts/2
label 10174 10148 0 15:22 pts/2 00:00:00 grep --color=auto sshd
[label@awx ~]$ taskset -p 1
pid 1's current affinity mask: f
[label@awx ~]$ taskset -p 896
pid 896's current affinity mask: f
[label@awx ~]$ taskset -p 1525
pid 1525's current affinity mask: f
[label@awx ~]$ taskset -p 1540
pid 1540's current affinity mask: f
[label@awx ~]$ taskset -p 10144
pid 10144's current affinity mask: f
[label@awx ~]$ taskset -p 10147
pid 10147's current affinity mask: f

When restarting sshd, I still cannot reproduce it.

[label@awx ~]$ sudo systemctl restart sshd
[sudo] password for label:
[label@awx ~]$ ps -aef | grep sshd
root 1525 1 0 Aug03 ? 00:00:00 sshd: label [priv]
label 1540 1525 0 Aug03 ? 00:00:00 sshd: label@pts/0
root 10144 1 0 15:22 ? 00:00:00 sshd: label [priv]
label 10147 10144 0 15:22 ? 00:00:00 sshd: label@pts/2
root 10194 1 0 15:25 ? 00:00:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
label 10196 10148 0 15:25 pts/2 00:00:00 grep --color=auto sshd
[label@awx ~]$ taskset -p 1
pid 1's current affinity mask: f
[label@awx ~]$ taskset -p 10194
pid 10194's current affinity mask: f
[label@awx ~]$ taskset -p 1525
pid 1525's current affinity mask: f
[label@awx ~]$ taskset -p 1540
pid 1540's current affinity mask: f
[label@awx ~]$ taskset -p 10144
pid 10144's current affinity mask: f
[label@awx ~]$ taskset -p 10147
pid 10147's current affinity mask: f

Setting to need info.
Roger Simmons

Roger Simmons

2022-08-12 07:24

reporter   ~0000338

Thanks Louis - can I double check if you changed the taskset for pid 1/init from the default "all cores" as the first step in testing.
The issue only shows when pid 1/init is set to only use a subset of cores.

If in step 1 you taskset init to a subset,
- eg: #> taskset 3 -p 1
- then restart sshd
- and ssh back onto the machine,
the child & grandchild pids will show as "all cores"/f rather then the 3 of init & sshd
Louis Abel

Louis Abel

2022-08-12 08:03

administrator   ~0000339

I did not modify the affinity before, that was a step I missed.

See below. The tests were performed on Rocky Linux 9, Rocky Linux 8, CentOS Stream 9

Rocky Linux 9

# Original affinity. Current pts/2 session.
[root@awx ~]# taskset -p 1
pid 1's current affinity mask: f
[root@awx ~]# ps -aef | grep sshd
root 1525 1 0 Aug03 ? 00:00:00 sshd: label [priv]
label 1540 1525 0 Aug03 ? 00:00:00 sshd: label@pts/0
root 10194 1 0 Aug11 ? 00:00:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
root 10694 10194 0 00:26 ? 00:00:00 sshd: label [priv]
label 10702 10694 0 00:26 ? 00:00:00 sshd: label@pts/2
root 10776 10728 0 00:41 pts/3 00:00:00 grep --color=auto sshd
[root@awx ~]# taskset -p 10194
pid 10194's current affinity mask: f
[root@awx ~]# taskset -p 10694
pid 10694's current affinity mask: f
[root@awx ~]# taskset -p 10702
pid 10702's current affinity mask: f

# Setting mask to 3
[root@awx ~]# taskset -p 3 1
pid 1's current affinity mask: f
pid 1's new affinity mask: 3
[root@awx ~]# taskset -p 1
pid 1's current affinity mask: 3
[root@awx ~]# taskset -c -p 1
pid 1's current affinity list: 0,1

# Restarting sshd
[root@awx ~]# systemctl restart sshd

# checking affinity has changed to 3 for new pts/2 session, while older pts/0 session is still f
[root@awx ~]# ps -aef | grep sshd
root 1525 1 0 Aug03 ? 00:00:00 sshd: label [priv]
label 1540 1525 0 Aug03 ? 00:00:00 sshd: label@pts/0
root 10786 1 0 00:43 ? 00:00:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
root 10788 10786 0 00:43 ? 00:00:00 sshd: label [priv]
label 10797 10788 0 00:43 ? 00:00:00 sshd: label@pts/2
root 10849 10824 0 00:44 pts/3 00:00:00 grep --color=auto sshd
[root@awx ~]# taskset -p 10786
pid 10786's current affinity mask: 3
[root@awx ~]# taskset -p 10788
pid 10788's current affinity mask: 3
[root@awx ~]# taskset -p 10797
pid 10797's current affinity mask: 3

---

CentOS Stream 9

# Current affinity
[root@localhost ~]# taskset -p 1
pid 1's current affinity mask: f

# Setting affinity to 3
[root@localhost ~]# taskset -p 3 1
pid 1's current affinity mask: f
pid 1's new affinity mask: 3
[root@localhost ~]# taskset -c -p 1
pid 1's current affinity list: 0,1
[root@localhost ~]# ps -aef | grep sshd
root 773 1 0 00:27 ? 00:00:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
root 1236 773 0 00:27 ? 00:00:00 sshd: root [priv]
root 1251 1236 0 00:27 ? 00:00:00 sshd: root@pts/0
root 1308 1252 0 00:31 pts/0 00:00:00 grep --color=auto sshd
[root@localhost ~]# taskset -p 1
pid 1's current affinity mask: 3

# Current sshd processes inherit the original mask of f
[root@localhost ~]# taskset -p 773
pid 773's current affinity mask: f
[root@localhost ~]# taskset -p 1236
pid 1236's current affinity mask: f
[root@localhost ~]# taskset -p 1251
pid 1251's current affinity mask: f

# The new sshd listener inherits 3
[root@localhost ~]# systemctl restart sshd
[root@localhost ~]# ps -aef | grep sshd
root 1236 1 0 00:27 ? 00:00:00 sshd: root [priv]
root 1251 1236 0 00:27 ? 00:00:00 sshd: root@pts/0
root 1316 1 0 00:32 ? 00:00:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
root 1319 1252 0 00:32 pts/0 00:00:00 grep --color=auto sshd
[root@localhost ~]# taskset -p 1316
pid 1316's current affinity mask: 3

# New sshd processes inherit 3
[root@localhost ~]# ps -aef | grep sshd
root 1236 1 0 00:27 ? 00:00:00 sshd: root [priv]
root 1251 1236 0 00:27 ? 00:00:00 sshd: root@pts/0
root 1316 1 0 00:32 ? 00:00:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
root 1324 1316 0 00:33 ? 00:00:00 sshd: root [priv]
root 1328 1324 0 00:33 ? 00:00:00 sshd: root@pts/1
root 1353 1329 0 00:33 pts/1 00:00:00 grep --color=auto sshd
[root@localhost ~]# taskset -p 1
pid 1's current affinity mask: 3
[root@localhost ~]# taskset -p 1316
pid 1316's current affinity mask: 3
[root@localhost ~]# taskset -p 1324
pid 1324's current affinity mask: 3
[root@localhost ~]# taskset -p 1328
pid 1328's current affinity mask: 3

---

Rocky Linux 8

[root@cm01 ~]# ps -aef | grep sshd
root 1007 1 0 Aug02 ? 00:00:00 /usr/sbin/sshd -D
root 4272 1007 0 Aug02 ? 00:00:00 sshd: label [priv]
label 4805 4272 0 Aug02 ? 00:00:00 sshd: label@pts/0
root 17519 1007 0 00:46 ? 00:00:00 sshd: label [priv]
label 17530 17519 0 00:46 ? 00:00:00 sshd: label@pts/2
root 17634 17605 0 00:46 pts/3 00:00:00 grep --color=auto sshd
[root@cm01 ~]# taskset -p 1
pid 1's current affinity mask: 3
[root@cm01 ~]# taskset -p 4272
pid 4272's current affinity mask: 3
[root@cm01 ~]# taskset -p 4805
pid 4805's current affinity mask: 3

[root@cm01 ~]# taskset -p 1 1
pid 1's current affinity mask: 3
pid 1's new affinity mask: 1
[root@cm01 ~]# systemctl restart sshd

# New session after restart. Affinity mask of 1 matches
[root@cm01 ~]# ps -aef | grep sshd
root 4272 1 0 Aug02 ? 00:00:00 sshd: label [priv]
label 4805 4272 0 Aug02 ? 00:00:00 sshd: label@pts/0
root 17648 1 0 00:48 ? 00:00:00 /usr/sbin/sshd -D
root 17662 17648 0 00:48 ? 00:00:00 sshd: label [priv]
label 17668 17662 0 00:48 ? 00:00:00 sshd: label@pts/2
root 17749 17717 0 00:49 pts/3 00:00:00 grep --color=auto sshd
[root@cm01 ~]# taskset -p 1
pid 1's current affinity mask: 1
[root@cm01 ~]# taskset -p 17648
pid 17648's current affinity mask: 1
[root@cm01 ~]# taskset -p 17662
pid 17662's current affinity mask: 1
[root@cm01 ~]# taskset -p 17668
pid 17668's current affinity mask: 1
Roger Simmons

Roger Simmons

2022-08-12 11:23

reporter   ~0000341

Thank you - that all looks completely normal.

Taking a further look at things this side I have tracked down the issue to an unexpected/unwanted cgroup setup.
Specifically /sys/fs/cgroup/cgroup.subtree_control contains cpuset, and this was interfering with the usual init/parent/child inheritance. Removing this returns the normal/expected behaviour

I have checked with a vanilla VM install of Rocky 9 and see that the default cgroup.subtree_control omits cpuset (and cpu) which is fine, so this must have been imposed by a 3rd party install somewhere along the way.

Thanks again. This can be closed; no issue.

Issue History

Date Modified Username Field Change
2022-08-11 21:57 Roger Simmons New Issue
2022-08-11 21:57 Roger Simmons Tag Attached: sshd
2022-08-11 21:57 Roger Simmons Tag Attached: taskset
2022-08-11 22:29 Louis Abel Assigned To => Louis Abel
2022-08-11 22:29 Louis Abel Status new => needinfo
2022-08-11 22:29 Louis Abel Note Added: 0000337
2022-08-12 07:24 Roger Simmons Note Added: 0000338
2022-08-12 08:03 Louis Abel Note Added: 0000339
2022-08-12 11:23 Roger Simmons Note Added: 0000341
2022-08-12 19:55 Louis Abel Assigned To Louis Abel =>
2022-08-12 19:55 Louis Abel Status needinfo => closed
2022-08-12 19:55 Louis Abel Resolution open => no change required