View Issue Details

IDProjectCategoryView StatusLast Update
0005611Rocky-Linux-9sssdpublic2024-12-02 08:41
ReporterDennis Brylow Assigned ToLouis Abel  
PriorityurgentSeveritymajorReproducibilityalways
Status closedResolutionfixed 
Platformx86_64OSRockyOS Version9.3
Fixed in Version9.5 
Summary0005611: sssd-2.9.1-4.el9_3.5 breaks authentication for all Active Directory users.
DescriptionDozens of machines lost the ability to authenticate any Active Directory (AD) users with the automatic upgrade to sssd-2.9.1-4.el9_3.5 and its associated packages this week.

Sshd reports in /var/log/secure:
sshd[420278]: pam_sss(sshd:account): Access denied for user foo: 4 (System error)
sshd[420278]: fatal: Access denied for user foo by PAM account configuration [preauth].

GUI does not authenticate.
Console prints "System error", and then reprompts for login.

Machines are still on the AD realm, and can pull info with 'getent passwd username'. Superuser can su to AD users and operate as normally. All local accounts appear to be functioning. But all AD accounts cannot authenticate, and the log messages are surprisingly terse and unhelpful.

Issue can be fixed by forcing downgrade back to sssd-2.9.1-4.el9_3.1.x86_64.rpm and 17 associated sssd-*, libsss_* and libipa_hbac packages.
Steps To Reproduce1) Install Minimal Rocky-9.3.
2) Install latest versions of realmd, sssd and associated co-requisites.
3) Join machine to AD realm and configure sssd.conf file.
4) Enjoy total absence of working AD authentication for all of your remote and console users.
Additional InformationAll prior versions of sssd have worked out-of-the-box in this Active Directory configuration since at least Rocky 8 and CentOS 7.

Full list of downgraded packages to effect solution:
libipa_hbac-2.9.1-4.el9_3.1.x86_64.rpm
libsss_autofs-2.9.1-4.el9_3.1.x86_64.rpm
libsss_certmap-2.9.1-4.el9_3.1.x86_64.rpm
libsss_idmap-2.9.1-4.el9_3.1.x86_64.rpm
libsss_nss_idmap-2.9.1-4.el9_3.1.x86_64.rpm
libsss_sudo-2.9.1-4.el9_3.1.x86_64.rpm
sssd-2.9.1-4.el9_3.1.x86_64.rpm
sssd-ad-2.9.1-4.el9_3.1.x86_64.rpm
sssd-client-2.9.1-4.el9_3.1.x86_64.rpm
sssd-common-2.9.1-4.el9_3.1.x86_64.rpm
sssd-common-pac-2.9.1-4.el9_3.1.x86_64.rpm
sssd-ipa-2.9.1-4.el9_3.1.x86_64.rpm
sssd-kcm-2.9.1-4.el9_3.1.x86_64.rpm
sssd-krb5-2.9.1-4.el9_3.1.x86_64.rpm
sssd-krb5-common-2.9.1-4.el9_3.1.x86_64.rpm
sssd-ldap-2.9.1-4.el9_3.1.x86_64.rpm
sssd-nfs-idmap-2.9.1-4.el9_3.1.x86_64.rpm
sssd-proxy-2.9.1-4.el9_3.1.x86_64.rpm
TagsNo tags attached.

Activities

Louis Abel

Louis Abel

2024-01-27 03:03

administrator   ~0005710

Thank you for the report. "4 (System error)" is an unhandled exception. This generally means that something is happening between SSSD and its auth provider (in this case AD) that it's not expecting and can't proceed.

With that said, I have been unable to reproduce this issue with this configuration:

* Windows Server 2022, Forest Level 2016, fully updated.
* Rocky Linux 9.3, fully updated. Enrolled with a standard configuration setup.

% dnf install realmd -y
% rpm -q sssd-ad realm
realmd-0.17.1-1.el9.x86_64
sssd-ad-2.9.1-4.el9_3.5.x86_64

% realm join ad.angelsofclockwork.net
Password for Administrator:
 * Installing necessary packages: oddjob-mkhomedir oddjob

% id label@AD.ANGELSOFCLOCKWORK.NET
uid=588201103(label@ad.angelsofclockwork.net) gid=588200513(domain users@ad.angelsofclockwork.net) groups=588200513(domain users@ad.angelsofclockwork.net)
% ssh label@ad.angelsofclockwork.net@localhost
label@ad.angelsofclockwork.net@localhost's password:
[label@ad.angelsofclockwork.net@rl9 ~]$ id
uid=588201103(label@ad.angelsofclockwork.net) gid=588200513(domain users@ad.angelsofclockwork.net) groups=588200513(domain users@ad.angelsofclockwork.net) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

GDM also works.

Jan 26 19:59:40 localhost gdm-password][37831]: pam_sss(gdm-password:auth): authentication success; logname= uid=0 euid=0 tty=/dev/tty1 ruser= rhost= user=label@ad.angelsofclockwork.net
Jan 26 19:59:40 localhost gdm-password][37831]: pam_unix(gdm-password:session): session opened for user label@ad.angelsofclockwork.net(uid=588201103) by (uid=0)

Please provide the following information:

* AD Domain Forest Level
* SSSD configuration - /etc/sssd/sssd.conf
* SSSD debug logs - See https://sssd.io/troubleshooting/basics.html for debugging steps.

Setting to needinfo.
Dennis Brylow

Dennis Brylow

2024-01-30 15:39

reporter   ~0005743

Thanks for taking a look at this with us, Louis.
   Domain and Forest functional level: Windows Server 2016.

sssd.conf:

--cut_here--
[sssd]
domains = mscsnet.mu.edu
config_file_version = 2
services = nss, pam

[nss]
enum_cache_timeout = 600
entry_cache_nowait_percentage = 60


[domain/mscsnet.mu.edu]
ad_domain = mscsnet.mu.edu
krb5_realm = MSCSNET.MU.EDU
realmd_tags = manages-system joined-with-samba
cache_credentials = True
entry_cache_timeout = 28800
id_provider = ad
krb5_store_password_if_offline = True
default_shell = /bin/bash
ldap_id_mapping = False
use_fully_qualified_names = False
access_provider = ad
ldap_schema = ad
auth_provider = ad
chpass_provider = ad
auto_private_groups = true
ldap_access_order = pwd_expire_policy_renew
dyndns_update_ptr = False
--cut_here--

var/log/secure error (one of many examples):
--cut_here--
Jan 26 13:53:17 npascal sshd[2066079]: pam_sss(sshd:auth): authentication success; logname= uid=0 euid=0 tty=ssh ruser=
rhost=134.48.5.61 user=testuser
Jan 26 13:53:17 npascal sshd[2066079]: pam_sss(sshd:account): Access denied for user testuser: 4 (System error)
Jan 26 13:53:17 npascal sshd[2066079]: Failed password for testuser from 134.48.5.61 port 49737 ssh2
Jan 26 13:53:17 npascal sshd[2066079]: fatal: Access denied for user testuser by PAM account configuration [preauth]
--cut_here--

I will grab some SSSD debugging logs for you this afternoon.
Dennis Brylow

Dennis Brylow

2024-03-01 19:39

reporter   ~0006271

Louis,
   No change here -- every time one of our Linux clients inadvertently upgrades to sssd-2.9.1-4.el9_3.5 and associated packages, all AD authentication breaks. No updates or bugfixes have been forthcoming from upstream that I can see.
   Our AD administrator says that our domain is actually Windows Server 2019, objectVersion = 88. We've started to see other reports of this problem popping up for others, but still no fixes. Our best guess is that something about our AD domain differs from whatever testing context you and others are using. But the AD domain has been the primary authentication source for both Windows and Linux clients here for at least four years.
   Please let me know if there is any more debugging info we can help provide to try to track this issue down.

Thanks!
Voja Molani

Voja Molani

2024-03-19 09:27

reporter   ~0006469

I may have been hitting this same issue as well. Possibly. The symptoms look very similar.

Updated from 2.9.1-4.el9_3 to 2.9.1-4.el9_3.5 and logins fail. GPO access control is in use.

The issue was tracked, using the SSSD logs, down to SSSD not finding the computer account of the machine being logged into.
It may be *partially* related to my configuration because `ldap_user_search_base` does *not* include computer accounts. Computer accounts are located in another OU not in this subtree.

Changing `ldap_user_search_base` to an OU that includes both computer + user accounts make everything work again, just like in 2.9.1-4.

It should be noted that specifying `ldap_host_search_base` to the subtree containing computer accounts did not help at all!

@Dennis Brylow, can you please check if in your case computer accounts are also outside of the "user" search base configured for SSSD?
Voja Molani

Voja Molani

2024-03-20 04:02

reporter   ~0006502

Not trying to hijack this but further investigation looks like I was on the correct track. I hope the OP could share some information on whether GPO access control is used.

My particular issue will be most likely resolved when https://github.com/SSSD/sssd/commit/29a77c6e79020d7e8cb474b4d3b394d390eba196 is picked to the RH SSSD release.

While looking at this I noticed another possibly relevant to the OP fix that's not yet in RH SSSD: `RHEL-22288 - ssh pubkey stored in ldap/AD no longer works to authenticate via sssd` -- is OP logging in with SSH pubkey stored in LDAP?
Louis Abel

Louis Abel

2024-11-27 01:44

administrator   ~0008917

Rocky Linux 9.5 has been released with version 2.9.5-4.el9_5.1. Please confirm if this issue still exists on this version.

Keeping "needinfo" status.
Dennis Brylow

Dennis Brylow

2024-12-01 22:42

reporter   ~0008944

Nope. Issue does *not* appear to exist in Rocky 9.5 with sssd-2.9.5-4.el9_5.1. Thanks for the attention!

@Voja Molani sorry I didn't see your question back in March, but yes, our computer accounts are stored outside of the user search base.
Louis Abel

Louis Abel

2024-12-02 08:41

administrator   ~0008977

Issue appears to be fixed in 9.5. Closing.

Issue History

Date Modified Username Field Change
2024-01-27 01:45 Dennis Brylow New Issue
2024-01-27 03:03 Louis Abel Assigned To => Louis Abel
2024-01-27 03:03 Louis Abel Status new => needinfo
2024-01-27 03:03 Louis Abel Note Added: 0005710
2024-01-30 15:39 Dennis Brylow Note Added: 0005743
2024-03-01 19:39 Dennis Brylow Note Added: 0006271
2024-03-19 09:27 Voja Molani Note Added: 0006469
2024-03-20 04:02 Voja Molani Note Added: 0006502
2024-11-27 01:44 Louis Abel Note Added: 0008917
2024-12-01 22:42 Dennis Brylow Note Added: 0008944
2024-12-02 08:41 Louis Abel Status needinfo => closed
2024-12-02 08:41 Louis Abel Resolution open => fixed
2024-12-02 08:41 Louis Abel Fixed in Version => 9.5
2024-12-02 08:41 Louis Abel Note Added: 0008977