Member since
07-16-2017
3
Posts
0
Kudos Received
0
Solutions
07-17-2017
02:15 PM
@Kuldeep Kulkarni, Thank you for your response. The user running the NFS Gateway appears to be the hdfs user per the following ps dump [root@smhadoop01 ~]# ps aux | grep nfs
root 3555 0.0 0.0 112648 960 pts/0 S+ 09:50 0:00 grep --color=auto nfs
root 10467 0.0 0.0 10688 884 ? S Jul16 0:00 jsvc.exec -Dproc_nfs3 -outfile /var/log/hadoop/root/nfs3_jsvc.out -errfile /var/log/hadoop/root/nfs3_jsvc.err -pidfile /var/run/hadoop/root/hadoop_privileged_nfs3.pid -nodetach -user hdfs -cp /usr/hdp/current/hadoop-client/conf:/usr/hdp/2.6.1.0-129/hadoop/lib/*:/usr/hdp/2.6.1.0-129/hadoop/.//*:/usr/hdp/2.6.1.0-129/hadoop-hdfs/./:/usr/hdp/2.6.1.0-129/hadoop-hdfs/lib/*:/usr/hdp/2.6.1.0-129/hadoop-hdfs/.//*:/usr/hdp/2.6.1.0-129/hadoop-yarn/lib/*:/usr/hdp/2.6.1.0-129/hadoop-yarn/.//*:/usr/hdp/2.6.1.0-129/hadoop-mapreduce/lib/*:/usr/hdp/2.6.1.0-129/hadoop-mapreduce/.//*::mysql-connector-java.jar:/usr/hdp/2.6.1.0-129/tez/*:/usr/hdp/2.6.1.0-129/tez/lib/*:/usr/hdp/2.6.1.0-129/tez/conf:mysql-connector-java.jar:mysql-connector-java.jar:/usr/hdp/2.6.1.0-129/tez/*:/usr/hdp/2.6.1.0-129/tez/lib/*:/usr/hdp/2.6.1.0-129/tez/conf -Xmx1024m -Dhdp.version=2.6.1.0-129 -Djava.net.preferIPv4Stack=true -Dhdp.version= -Djava.net.preferIPv4Stack=true -Dhdp.version= -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/var/log/hadoop/ -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.6.1.0-129/hadoop -Dhadoop.id.str= -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/2.6.1.0-129/hadoop/lib/native/Linux-amd64-64:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.6.1.0-129/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dhdp.version=2.6.1.0-129 -Dhadoop.log.dir=/var/log/hadoop/ -Dhadoop.log.file=hadoop-hdfs-nfs3-smhadoop01.na.corning.com.log -Dhadoop.home.dir=/usr/hdp/2.6.1.0-129/hadoop -Dhadoop.id.str= -Dhadoop.root.logger=INFO,RFA -Djava.library.path=:/usr/hdp/2.6.1.0-129/hadoop/lib/native/Linux-amd64-64:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.6.1.0-129/hadoop/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.6.1.0-129/hadoop/lib/native/Linux-amd64-64:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.6.1.0-129/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/var/log/hadoop/root -Dhadoop.id.str=hdfs -Xmx1024m -Dhadoop.security.logger=ERROR,DRFAS -Dhadoop.security.logger=INFO,RFAS org.apache.hadoop.hdfs.nfs.nfs3.PrivilegedNfsGatewayStarter
hdfs 10498 0.2 0.3 2898548 496400 ? Sl Jul16 2:54 jsvc.exec -Dproc_nfs3 -outfile /var/log/hadoop/root/nfs3_jsvc.out -errfile /var/log/hadoop/root/nfs3_jsvc.err -pidfile /var/run/hadoop/root/hadoop_privileged_nfs3.pid -nodetach -user hdfs -cp /usr/hdp/current/hadoop-client/conf:/usr/hdp/2.6.1.0-129/hadoop/lib/*:/usr/hdp/2.6.1.0-129/hadoop/.//*:/usr/hdp/2.6.1.0-129/hadoop-hdfs/./:/usr/hdp/2.6.1.0-129/hadoop-hdfs/lib/*:/usr/hdp/2.6.1.0-129/hadoop-hdfs/.//*:/usr/hdp/2.6.1.0-129/hadoop-yarn/lib/*:/usr/hdp/2.6.1.0-129/hadoop-yarn/.//*:/usr/hdp/2.6.1.0-129/hadoop-mapreduce/lib/*:/usr/hdp/2.6.1.0-129/hadoop-mapreduce/.//*::mysql-connector-java.jar:/usr/hdp/2.6.1.0-129/tez/*:/usr/hdp/2.6.1.0-129/tez/lib/*:/usr/hdp/2.6.1.0-129/tez/conf:mysql-connector-java.jar:mysql-connector-java.jar:/usr/hdp/2.6.1.0-129/tez/*:/usr/hdp/2.6.1.0-129/tez/lib/*:/usr/hdp/2.6.1.0-129/tez/conf -Xmx1024m -Dhdp.version=2.6.1.0-129 -Djava.net.preferIPv4Stack=true -Dhdp.version= -Djava.net.preferIPv4Stack=true -Dhdp.version= -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/var/log/hadoop/ -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.6.1.0-129/hadoop -Dhadoop.id.str= -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/2.6.1.0-129/hadoop/lib/native/Linux-amd64-64:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.6.1.0-129/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dhdp.version=2.6.1.0-129 -Dhadoop.log.dir=/var/log/hadoop/ -Dhadoop.log.file=hadoop-hdfs-nfs3-smhadoop01.na.corning.com.log -Dhadoop.home.dir=/usr/hdp/2.6.1.0-129/hadoop -Dhadoop.id.str= -Dhadoop.root.logger=INFO,RFA -Djava.library.path=:/usr/hdp/2.6.1.0-129/hadoop/lib/native/Linux-amd64-64:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.6.1.0-129/hadoop/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.6.1.0-129/hadoop/lib/native/Linux-amd64-64:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.6.1.0-129/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/var/log/hadoop/root -Dhadoop.id.str=hdfs -Xmx1024m -Dhadoop.security.logger=ERROR,DRFAS -Dhadoop.security.logger=INFO,RFAS org.apache.hadoop.hdfs.nfs.nfs3.PrivilegedNfsGatewayStarter And the core-site.xml settings seem to reflect the values you are looking for [root@smhadoop01 ~]# grep -B1 -A2 proxyuser\.hdfs /etc/hadoop/conf/core-site.xml
<property>
<name>hadoop.proxyuser.hdfs.groups</name>
<value>*</value>
</property>
--
<property>
<name>hadoop.proxyuser.hdfs.hosts</name>
<value>*</value>
</property>
And I am able to mount the NFS export on the edge node if I don't enable Kerberos security [root@smhadoop-edge ~]# mount smhadoop01:/ /mnt
[root@smhadoop-edge ~]# mount | grep smhadoop01
smhadoop01:/ on /mnt type nfs (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.180.104.161,mountvers=3,mountport=4242,mountproto=udp,local_lock=none,addr=10.180.104.161)
[root@smhadoop-edge ~]# umount /mnt
[root@smhadoop-edge ~]# mount -v -o sec=krb5,noatime,nolock smhadoop01:/ /mnt
mount.nfs: timeout set for Mon Jul 17 09:47:26 2017
mount.nfs: trying text-based options 'sec=krb5,nolock,vers=4,addr=10.180.104.161,clientaddr=10.180.104.38'
mount.nfs: mount(2): Protocol not supported
mount.nfs: trying text-based options 'sec=krb5,nolock,addr=10.180.104.161'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 10.180.104.161 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 10.180.104.161 prog 100005 vers 3 prot UDP port 4242
mount.nfs: mount(2): Permission denied
mount.nfs: access denied by server while mounting smhadoop01:/
[root@smhadoop-edge ~]#
This really appears to be a problem negotiating Kerberos security on the share.
... View more
07-17-2017
02:14 PM
Sorry, I am still working out this forums mechanics and I posted an answer when I meant to post a Reply.
... View more
07-16-2017
05:06 PM
I am getting an error similar to the one asked about in the question "Problems with Kerberos (NFS and File View)" except my File View and Hadoop fs commands work fine with Kerberos. I have used the Kerberos Wizard and checks completed with no errors. I double checked the settings of the NFS Gateway to ensure it was correctly configured by the Wizard and it matched the instructions for the version of HDP I am running (2.6.1). Here is what I get from the edge node when attempting a mount [root@smhadoop-edge ~]# klist klist: No credentials cache found (filename: /tmp/krb5cc_0)
[root@smhadoop-edge ~]# kinit WilsonC
Password for WilsonC@NA.CORNING.COM:
[root@smhadoop-edge ~]# klist
Ticket cache: FILE:/tmp/krb5cc_0 Default principal: WilsonC@NA.CORNING.COM
Valid starting Expires Service principal 07/16/2017 12:59:11 07/16/2017 22:59:11 krbtgt/NA.CORNING.COM@NA.CORNING.COM renew until 07/16/2017 22:59:11
[root@smhadoop-edge ~]# mount -vvv -o nolock,sec=krb5,noatime smhadoop01:/ /mnt
mount.nfs: timeout set for Sun Jul 16 13:01:39 2017
mount.nfs: trying text-based options 'nolock,sec=krb5,vers=4,addr=10.180.104.161,clientaddr=10.180.104.38'
mount.nfs: mount(2): Protocol not supported
mount.nfs: trying text-based options 'nolock,sec=krb5,addr=10.180.104.161'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 10.180.104.161 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 10.180.104.161 prog 100005 vers 3 prot UDP port 4242
mount.nfs: mount(2): Permission denied
mount.nfs: access denied by server while mounting smhadoop01:/
However the normal Hadoop commands work fine with Kerberos [root@smhadoop-edge ~]# hadoop fs -mkdir /user/WilsonC/testing
[root@smhadoop-edge ~]# hadoop fs -ls /user/WilsonC
Found 1 items drwxr-xr-x - WilsonC hdfs 0 2017-07-16 13:02 /user/WilsonC/testing
Any help is much appreciated. I am also working on getting Kerberos to work correctly with NFS on Windows if anybody has any pointers.
... View more
Labels:
