I had the weirdest thing suddenly start happening last night that took several hours to finally figure out was a time-related issue.
I've got an Ubuntu box that uses
pam_winbind to allow for logging into a machine using an Active Directory account.
Normally I connect with an SSH key, but once in when doing
sudo -s I enter an AD password to become root. Last night
sudo -s suddenly stopped working.
Luckily I had another non-AD account that I could connect with, and sudo worked for that, so I could become root and poke around. The logs showed:
sudo: pam_unix(sudo:auth): authentication failure; logname=barry.pederson uid=14283 euid=0 tty=/dev/pts/0 ruser=barry.pederson rhost= user=barry.pederson sudo: pam_unix(sudo:auth): conversation failed sudo: pam_unix(sudo:auth): auth could not identify password for [barry.pederson]
That was weird, I could log into other things though that used the same AD account, so I knew the password was right and the account wasn't locked out.
I hoped by the next morning, some cache thing would expire and I'd be back in business, but no dice.
Poking around some more I found if I disabled my SSH keys, I couldn't log in at all, so it was really a
pam_winbind issue, not a
sudo one. The logs for a SSH password login attempt were a bit more informative:
pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=xxx.yyy.zzz user=barry.pederson pam_winbind(sshd:auth): getting password (0x00000388) pam_winbind(sshd:auth): pam_get_item returned a password pam_winbind(sshd:auth): request wbcLogonUser failed: WBC_ERR_AUTH_ERROR, PAM error: PAM_AUTH_ERR (7), NTSTATUS: NT_STATUS_LOGON_FAILURE, Error message was: Logon failure pam_winbind(sshd:auth): user 'barry.pederson' denied access (incorrect password or invalid membership) Failed password for barry.pederson from x.x.x.x port 50655 ssh2
WTF? I know the password's right, I've been typing it all morning into other systems. I even tried
wbinfo --authenticate barry.pederson on this box and it accepted my passwords.
Much time was spent Googling, trying various tweaks to
smb.conf, etc. Finally, I don't remember why, I thought to check the date with
ntpdate -d my.ad.server and it came back with
offset -338.308573 sec. Holy crap, that's more than 5 minutes! Even though ntpd is running.
Anyhow, once the clock was fixed to be closer to the AD server, logins and sudo started working again.