Unable to authenticate after upgrading

Do you have a question? Post it now! No Registration Necessary.  Now with pictures!

Threaded View
I just finished upgrading an AIX server from 5.1.03 to 5.3.01.  I also
upgraded openSSH from 3.8p1 (I think), compiled locally, to 4.1p1,
downloaded from IBM's "OpenSSH on AIX" Sourceforge site.  I kept the
same options in the ssh*_config files and copied the host keys from the
old location (/usr/local/etc) to the new (/etc/ssh).  All users kept
their .ssh directories.

I have one account that I set up several montha ago with a DSA key
pair, no passphrase, for running a batch job from another server - scp
followed by ssh with remote command execution.  If it matters, it's a
ksh cgi-bin script run under an Apache web server as "nobody",
connecting to the "acsss" account on an ACSLS server.  It worked before
the upgrade but is broken now.  I see the following error in the syslog
when the script tries to run, and SSH drops back to asking me for the
remote account's password.

Nov 20 02:19:07 <server_name> sshd[9814]: Authentication refused:
realpath /export/home/ACSSS/.ssh/authorized_keys failed: Permission

Here is the ssh invocation:
ssh -i /.ssh/id_dsa acsss@<server_name> <script_name>
(Yes, the key file is in root's ssh directory, but / is also "nobody"'s
home directory, the key file is owned by "nobody", "nobody" has read
access to the directory, and most importantly, it worked before.  I've
also run the command as root, with the same results, and running with
the -vvv verbose option shows that the key is sent and rejected, so I
don't think it's a problem on the sending side.)

Here are the relevant authorized_keys file and directory permissions:

#ls -ld /export /export/home /export/home/ACSSS \
    /export/home/ACSSS/.ssh/ /export/home/ACSSS/.ssh/authorized_keys
drwxr-xr-x   4 root     system   512 Nov 17 21:11 /export
drwxr-xr-x   6 sys      sys      512 Dec 14 2004  /export/home
drwxr-x---  15 acsss    staff   1024 Nov 17 22:15 /export/home/ACSSS
drwxr-x---   2 acsss    staff    512 Nov 18 16:33
-rw-r-----   1 acsss    staff    215 Nov 18 15:56

I have no problem logging in myself.  Here are my file/directory

#ls -ld /home /home/<my_id> /home/<my_id>/.ssh \
drwxr-xr-x  31 root     system   512 Nov 18 09:25 /home
drwxr-x---   4 <my_id>  staff    512 Nov 21 07:34 /home/<my_id>
drwxr-x---   2 <my_id>  staff    512 Aug 29 2003  /home/<my_id>/.ssh
-rw-r-----   1 <my_id>  staff    290 Oct 12 2001

I generated and tried a protocol 2 RSA key, but it shows the same
problem.  What does this error mean, or how can I get more detail?

Note: I can't see how it would make a difference, but somehow during
the AIX upgrade the /export/home mount point was lost.  I recreated it
with a simple mkdir and the filesystem mounted successfully.  Could the
mount point permisions be a problem, even though they're masked once
the filesystem is mounted?


Re: Unable to authenticate after upgrading

OK, never mind.  It is, of course, the one thing that I thought it
couldn't be.  We happened to have a scheduled outage today for
something else, so I took the opportunity to umount the home directory
and check the permissions.  The world r & x bits were indeed not set,
and as soon as I set them and remounted the filesystem, everything

When I recreated the home directory mount point the other day, I had
our normal umask of 027 in place, rather than the default (and
insecure) 022, so the new directory was not world-readable or -CD'able
(so to speak).  And even though the permissions were not visible, they
did prevent read access, so the keys were not available.

Now, the last remaining questoin is how do I get the blood off the wall
where I've been banging my head for the last three days?


Re: Unable to authenticate after upgrading

Quoted text here. Click to load it

I find that painting the wall is sometimes easier. But I'm confused. You saw
permissions of the mount point, *AFTER* you mounted a directory on it? That
makes no sense!

Re: Unable to authenticate after upgrading

I could only SEE the permissions when the filesystem was unmounted, but
they still affected the permissions of the filesystem AFTER it was
remounted.  I know it makes no sense, and I would never have even
considered it if I hadn't actually seen it before, when doing an
upgrade of AIX using the alt_disk_install process.  (For those
unfamiliar with AIX, this allows you to "clone" the root drive to
another drive and upgrade the cloned system, so you can subsequently
boot to the alternate disk and minimize the downtime for the upgrade).
During the initial cloning process, I need to be careful to set my
umask back to the default 022, otherwise the cloned filesystems'
permissions are affected and the system will not reboot.  It drove me
nuts until I unmounted the filesystems and saw that the mount point
directories had no world read or execute access.  One I set those bits,
everything worked fine.  Same thing here.

Re: Unable to authenticate after upgrading

Quoted text here. Click to load it

Yes, this is a famous problem with nfs and unix. BOTH the mount point's and
the mounted system's permissions are used to determine access. Of course the
mount point's permissions are invisible after the filesystem is mounted.

Re: Unable to authenticate after upgrading

Quoted text here. Click to load it

Ouch. I could have sworn that the mounted directories permissions where the
only ones used.

Site Timeline