Winbind failure do to incorrect time

Wednesday, July 8, 2015

I had the weirdest thing suddenly start happening last night that took several hours to finally figure out was a time-related issue.

I've got an Ubuntu box that uses pam_winbind to allow for logging into a machine using an Active Directory account.
Normally I connect with an SSH key, but once in when doing sudo -s I enter an AD password to become root. Last night that sudo -s suddenly stopped working.

Luckily I had another non-AD account that I could connect with, and sudo worked for that, so I could become root and poke around. The logs showed:

sudo: pam_unix(sudo:auth): authentication failure; logname=barry.pederson uid=14283 euid=0 tty=/dev/pts/0 ruser=barry.pederson rhost=  user=barry.pederson
sudo: pam_unix(sudo:auth): conversation failed
sudo: pam_unix(sudo:auth): auth could not identify password for [barry.pederson]

That was weird, I could log into other things though that used the same AD account, so I knew the password was right and the account wasn't locked out.

I hoped by the next morning, some cache thing would expire and I'd be back in business, but no dice.

Poking around some more I found if I disabled my SSH keys, I couldn't log in at all, so it was really a pam_winbind issue, not a sudo one. The logs for a SSH password login attempt were a bit more informative:

pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=xxx.yyy.zzz  user=barry.pederson
pam_winbind(sshd:auth): getting password (0x00000388)
pam_winbind(sshd:auth): pam_get_item returned a password
pam_winbind(sshd:auth): request wbcLogonUser failed: WBC_ERR_AUTH_ERROR, PAM error: PAM_AUTH_ERR (7), NTSTATUS: NT_STATUS_LOGON_FAILURE, Error message was: Logon failure
pam_winbind(sshd:auth): user 'barry.pederson' denied access (incorrect password or invalid membership)
Failed password for barry.pederson from x.x.x.x port 50655 ssh2

WTF? I know the password's right, I've been typing it all morning into other systems. I even tried wbinfo --authenticate barry.pederson on this box and it accepted my passwords.

Much time was spent Googling, trying various tweaks to smb.conf, etc. Finally, I don't remember why, I thought to check the date with ntpdate -d my.ad.server and it came back with offset -338.308573 sec. Holy crap, that's more than 5 minutes! Even though ntpd is running.

Anyhow, once the clock was fixed to be closer to the AD server, logins and sudo started working again.