Mounting NFS from Edge Node with Kerberos

[#1]

I am getting an error similar to the one asked about in the question "Problems with Kerberos (NFS and File View)" except my File View and Hadoop fs commands work fine with Kerberos. I have used the Kerberos Wizard and checks completed with no errors. I double checked the settings of the NFS Gateway to ensure it was correctly configured by the Wizard and it matched the instructions for the version of HDP I am running (2.6.1).

Here is what I get from the edge node when attempting a mount

[root@smhadoop-edge ~]# klist klist: No credentials cache found (filename: /tmp/krb5cc_0)
[root@smhadoop-edge ~]# kinit WilsonC 
Password for WilsonC@NA.CORNING.COM:
[root@smhadoop-edge ~]# klist
Ticket cache: FILE:/tmp/krb5cc_0 Default principal: WilsonC@NA.CORNING.COM
Valid starting  Expires  Service principal 07/16/2017 12:59:11  07/16/2017 22:59:11  krbtgt/NA.CORNING.COM@NA.CORNING.COM  renew until 07/16/2017 22:59:11 
[root@smhadoop-edge ~]# mount -vvv -o nolock,sec=krb5,noatime smhadoop01:/ /mnt 
mount.nfs: timeout set for Sun Jul 16 13:01:39 2017 
mount.nfs: trying text-based options 'nolock,sec=krb5,vers=4,addr=10.180.104.161,clientaddr=10.180.104.38' 
mount.nfs: mount(2): Protocol not supported 
mount.nfs: trying text-based options 'nolock,sec=krb5,addr=10.180.104.161' 
mount.nfs: prog 100003, trying vers=3, prot=6 
mount.nfs: trying 10.180.104.161 prog 100003 vers 3 prot TCP port 2049 
mount.nfs: prog 100005, trying vers=3, prot=17 
mount.nfs: trying 10.180.104.161 prog 100005 vers 3 prot UDP port 4242 
mount.nfs: mount(2): Permission denied
mount.nfs: access denied by server while mounting smhadoop01:/

However the normal Hadoop commands work fine with Kerberos

[root@smhadoop-edge ~]# hadoop fs -mkdir /user/WilsonC/testing 
[root@smhadoop-edge ~]# hadoop fs -ls /user/WilsonC 
Found 1 items drwxr-xr-x  - WilsonC hdfs  0 2017-07-16 13:02 /user/WilsonC/testing

Any help is much appreciated. I am also working on getting Kerberos to work correctly with NFS on Windows if anybody has any pointers.

[#2]

@Christopher Wilson

Please make sure that you have set hadoop.proxyuser.<nfs-user>.hosts property in core-site.xml to '*' or you have your hostname as a part of value of this property from where you are trying to mount NFS.

[#3]

Sorry, I am still working out this forums mechanics and I posted an answer when I meant to post a Reply.

[#4]

@Kuldeep Kulkarni, Thank you for your response.

The user running the NFS Gateway appears to be the hdfs user per the following ps dump

[root@smhadoop01 ~]# ps aux | grep nfs
root      3555  0.0  0.0 112648   960 pts/0    S+   09:50   0:00 grep --color=auto nfs
root     10467  0.0  0.0  10688   884 ?        S    Jul16   0:00 jsvc.exec -Dproc_nfs3 -outfile /var/log/hadoop/root/nfs3_jsvc.out -errfile /var/log/hadoop/root/nfs3_jsvc.err -pidfile /var/run/hadoop/root/hadoop_privileged_nfs3.pid -nodetach -user hdfs -cp /usr/hdp/current/hadoop-client/conf:/usr/hdp/2.6.1.0-129/hadoop/lib/*:/usr/hdp/2.6.1.0-129/hadoop/.//*:/usr/hdp/2.6.1.0-129/hadoop-hdfs/./:/usr/hdp/2.6.1.0-129/hadoop-hdfs/lib/*:/usr/hdp/2.6.1.0-129/hadoop-hdfs/.//*:/usr/hdp/2.6.1.0-129/hadoop-yarn/lib/*:/usr/hdp/2.6.1.0-129/hadoop-yarn/.//*:/usr/hdp/2.6.1.0-129/hadoop-mapreduce/lib/*:/usr/hdp/2.6.1.0-129/hadoop-mapreduce/.//*::mysql-connector-java.jar:/usr/hdp/2.6.1.0-129/tez/*:/usr/hdp/2.6.1.0-129/tez/lib/*:/usr/hdp/2.6.1.0-129/tez/conf:mysql-connector-java.jar:mysql-connector-java.jar:/usr/hdp/2.6.1.0-129/tez/*:/usr/hdp/2.6.1.0-129/tez/lib/*:/usr/hdp/2.6.1.0-129/tez/conf -Xmx1024m -Dhdp.version=2.6.1.0-129 -Djava.net.preferIPv4Stack=true -Dhdp.version= -Djava.net.preferIPv4Stack=true -Dhdp.version= -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/var/log/hadoop/ -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.6.1.0-129/hadoop -Dhadoop.id.str= -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/2.6.1.0-129/hadoop/lib/native/Linux-amd64-64:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.6.1.0-129/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dhdp.version=2.6.1.0-129 -Dhadoop.log.dir=/var/log/hadoop/ -Dhadoop.log.file=hadoop-hdfs-nfs3-smhadoop01.na.corning.com.log -Dhadoop.home.dir=/usr/hdp/2.6.1.0-129/hadoop -Dhadoop.id.str= -Dhadoop.root.logger=INFO,RFA -Djava.library.path=:/usr/hdp/2.6.1.0-129/hadoop/lib/native/Linux-amd64-64:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.6.1.0-129/hadoop/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.6.1.0-129/hadoop/lib/native/Linux-amd64-64:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.6.1.0-129/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/var/log/hadoop/root -Dhadoop.id.str=hdfs -Xmx1024m -Dhadoop.security.logger=ERROR,DRFAS -Dhadoop.security.logger=INFO,RFAS org.apache.hadoop.hdfs.nfs.nfs3.PrivilegedNfsGatewayStarter
hdfs     10498  0.2  0.3 2898548 496400 ?      Sl   Jul16   2:54 jsvc.exec -Dproc_nfs3 -outfile /var/log/hadoop/root/nfs3_jsvc.out -errfile /var/log/hadoop/root/nfs3_jsvc.err -pidfile /var/run/hadoop/root/hadoop_privileged_nfs3.pid -nodetach -user hdfs -cp /usr/hdp/current/hadoop-client/conf:/usr/hdp/2.6.1.0-129/hadoop/lib/*:/usr/hdp/2.6.1.0-129/hadoop/.//*:/usr/hdp/2.6.1.0-129/hadoop-hdfs/./:/usr/hdp/2.6.1.0-129/hadoop-hdfs/lib/*:/usr/hdp/2.6.1.0-129/hadoop-hdfs/.//*:/usr/hdp/2.6.1.0-129/hadoop-yarn/lib/*:/usr/hdp/2.6.1.0-129/hadoop-yarn/.//*:/usr/hdp/2.6.1.0-129/hadoop-mapreduce/lib/*:/usr/hdp/2.6.1.0-129/hadoop-mapreduce/.//*::mysql-connector-java.jar:/usr/hdp/2.6.1.0-129/tez/*:/usr/hdp/2.6.1.0-129/tez/lib/*:/usr/hdp/2.6.1.0-129/tez/conf:mysql-connector-java.jar:mysql-connector-java.jar:/usr/hdp/2.6.1.0-129/tez/*:/usr/hdp/2.6.1.0-129/tez/lib/*:/usr/hdp/2.6.1.0-129/tez/conf -Xmx1024m -Dhdp.version=2.6.1.0-129 -Djava.net.preferIPv4Stack=true -Dhdp.version= -Djava.net.preferIPv4Stack=true -Dhdp.version= -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/var/log/hadoop/ -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.6.1.0-129/hadoop -Dhadoop.id.str= -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/2.6.1.0-129/hadoop/lib/native/Linux-amd64-64:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.6.1.0-129/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dhdp.version=2.6.1.0-129 -Dhadoop.log.dir=/var/log/hadoop/ -Dhadoop.log.file=hadoop-hdfs-nfs3-smhadoop01.na.corning.com.log -Dhadoop.home.dir=/usr/hdp/2.6.1.0-129/hadoop -Dhadoop.id.str= -Dhadoop.root.logger=INFO,RFA -Djava.library.path=:/usr/hdp/2.6.1.0-129/hadoop/lib/native/Linux-amd64-64:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.6.1.0-129/hadoop/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.6.1.0-129/hadoop/lib/native/Linux-amd64-64:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.6.1.0-129/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/var/log/hadoop/root -Dhadoop.id.str=hdfs -Xmx1024m -Dhadoop.security.logger=ERROR,DRFAS -Dhadoop.security.logger=INFO,RFAS org.apache.hadoop.hdfs.nfs.nfs3.PrivilegedNfsGatewayStarter

And the core-site.xml settings seem to reflect the values you are looking for

[root@smhadoop01 ~]# grep -B1 -A2 proxyuser\.hdfs /etc/hadoop/conf/core-site.xml
    <property>
      <name>hadoop.proxyuser.hdfs.groups</name>
      <value>*</value>
    </property>
--
    <property>
      <name>hadoop.proxyuser.hdfs.hosts</name>
      <value>*</value>
    </property>

And I am able to mount the NFS export on the edge node if I don't enable Kerberos security

[root@smhadoop-edge ~]# mount smhadoop01:/ /mnt
[root@smhadoop-edge ~]# mount | grep smhadoop01
smhadoop01:/ on /mnt type nfs (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.180.104.161,mountvers=3,mountport=4242,mountproto=udp,local_lock=none,addr=10.180.104.161)
[root@smhadoop-edge ~]# umount /mnt
[root@smhadoop-edge ~]# mount -v -o sec=krb5,noatime,nolock smhadoop01:/ /mnt
mount.nfs: timeout set for Mon Jul 17 09:47:26 2017
mount.nfs: trying text-based options 'sec=krb5,nolock,vers=4,addr=10.180.104.161,clientaddr=10.180.104.38'
mount.nfs: mount(2): Protocol not supported
mount.nfs: trying text-based options 'sec=krb5,nolock,addr=10.180.104.161'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 10.180.104.161 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 10.180.104.161 prog 100005 vers 3 prot UDP port 4242
mount.nfs: mount(2): Permission denied
mount.nfs: access denied by server while mounting smhadoop01:/
[root@smhadoop-edge ~]#

This really appears to be a problem negotiating Kerberos security on the share.