Sample MR job fails from edge node after encrypting user home folder in HDFS on kerberized cluster

[#1]

I'm having issues validating my Kerberos + HDFS TDE setup on CDH 5.7.1. I'm running sample MapReduce job (Pi number calculation) that fails with Kerberos ticket error (claiming user has no valid credentials) when initiated from edge node only, after encrypting (HDFS Transparent Data Encryption) user home folder (/user/sampleuser) in HDFS on kerberized cluster. User executing job authenticated (via kinit) as cluster local MIT KDC principal (sampleuser@LOCALMITKDCREALM.COMPANY.COM). Same user, with same Kerberos ticket,  can succesfully connect to Hive (via beeline) and run HiveQL commands, as well as run hdfs dfs -ls /user/sampleuser or hdfs dfs -put tmp/localfile /user/sampleuser/hdfsdestfile or hdfs dfs -get /user/sampleuser/hdfsdestfile tmp/localfile, from the same edge node where I'm having issue with sample Pi MR job mentioned above (which suggests that Kerberos and HDFS TDE must be working for HDFS and Hive). Also, same user, authenticated (via kinit) as cluster local MIT KDC principal, can succesfully run the same sample MapReduce job (Pi number calculation) from any other node of the cluster. Users that do not have their home HDFS folder encrypted with HDFS TDE have no issues executing sample MR Pi job from either edge node or any other nodes on the cluster.

My environment is as follows:

- CDH 5.7.1, secured with local MIT KDC with one way trust to Active Directory KDC.

- Edge node:

1. Is managed by Cloudera Cluster Manager.
2. Latest cluster CDH client and Kerberos configs have been deployed to edge node.
3. JCE Unlimited Strength Jurisdiction Policy Files have been deployed to edge and JDK 1.8.0_92 is matching the rest of the cluster.
4. HDFS transparent data encryption is enabled using on-cluster Cloudera Navigator Key Trustee Server / Key Trustee KMS server.

User executing user has full access to the encryption keys for his HDFS home folder encryption zone (no custom KMS ACLs have been deloyed). Error suggests that no valid Kerberos credentials have been provided which is not true. Please advise if you have encountered this or have any thoughts. At this point I'm ready to give up on Kerberos + Encrypted (TDE) HDFS home user folder configuration as non-working in case of edge node as source of MR job submission. 

$ kinit sampleuser@LOCALMITKDCREALM.COMPANY.COM
Password for sampleuser@LOCALMITKDCREALM.COMPANY.COM:
$ klist
Ticket cache: FILE:/tmp/krb5cc_10048
Default principal: sampleuser@LOCALMITKDCREALM.COMPANY.COM

Valid starting Expires Service principal
12/18/16 22:06:29 12/19/16 22:06:29 krbtgt/LOCALMITKDCREALM.COMPANY.COM@LOCALMITKDCREALM.COMPANY.COM
renew until 12/25/16 22:06:29
$ hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop-0.20-mapreduce/hadoop-examples.jar pi 10 10000
Number of Maps = 10
Samples per Map = 10000
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Wrote input for Map #4
Wrote input for Map #5
Wrote input for Map #6
Wrote input for Map #7
Wrote input for Map #8
Wrote input for Map #9
Starting Job
16/12/18 22:09:59 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 294 for sampleuser on ha-hdfs:CDHCLUSTER-ns
16/12/18 22:09:59 INFO security.TokenCache: Got dt for hdfs://CDHCLUSTER-ns; Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:CDHCLUSTER-ns, Ident: (HDFS_DELEGATION_TOKEN token 294 for sampleuser)
16/12/18 22:09:59 WARN token.Token: Cannot find class for token kind kms-dt
16/12/18 22:09:59 INFO security.TokenCache: Got dt for hdfs://CDHCLUSTER-ns; Kind: kms-dt, Service: xx.xxx.xxx.16:16000, Ident: 00 03 62 64 64 04 79 61 72 6e 00 8a 01 59 15 45 98 c3 8a 01 59 39 52 1c c3 8e 01 12 7c
16/12/18 22:09:59 WARN token.Token: Cannot find class for token kind kms-dt
16/12/18 22:09:59 INFO security.TokenCache: Got dt for hdfs://CDHCLUSTER-ns; Kind: kms-dt, Service: xx.xxx.xxx.15:16000, Ident: 00 03 62 64 64 04 79 61 72 6e 00 8a 01 59 15 45 99 4d 8a 01 59 39 52 1d 4d 8e 01 13 7a
16/12/18 22:10:00 INFO input.FileInputFormat: Total input paths to process : 10
16/12/18 22:10:00 INFO mapreduce.JobSubmitter: number of splits:10
16/12/18 22:10:00 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1481262207801_0093
16/12/18 22:10:00 INFO mapreduce.JobSubmitter: Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:CDHCLUSTER-ns, Ident: (HDFS_DELEGATION_TOKEN token 294 for sampleuser)
16/12/18 22:10:00 WARN token.Token: Cannot find class for token kind kms-dt
16/12/18 22:10:00 WARN token.Token: Cannot find class for token kind kms-dt
Kind: kms-dt, Service: xx.xxx.xxx.15:16000, Ident: 00 03 62 64 64 04 79 61 72 6e 00 8a 01 59 15 45 99 4d 8a 01 59 39 52 1d 4d 8e 01 13 7a
16/12/18 22:10:00 WARN token.Token: Cannot find class for token kind kms-dt
16/12/18 22:10:00 WARN token.Token: Cannot find class for token kind kms-dt
Kind: kms-dt, Service: xx.xxx.xxx.16:16000, Ident: 00 03 62 64 64 04 79 61 72 6e 00 8a 01 59 15 45 98 c3 8a 01 59 39 52 1c c3 8e 01 12 7c
16/12/18 22:10:00 INFO impl.YarnClientImpl: Submitted application application_1481262207801_0093
16/12/18 22:10:00 INFO mapreduce.Job: The url to track the job: http://CDHCLUSTERn03.COMPANY.COM:8088/proxy/application_1481262207801_0093/
16/12/18 22:10:00 INFO mapreduce.Job: Running job: job_1481262207801_0093
16/12/18 22:10:05 INFO mapreduce.Job: Job job_1481262207801_0093 running in uber mode : false
16/12/18 22:10:05 INFO mapreduce.Job: map 0% reduce 0%
16/12/18 22:10:05 INFO mapreduce.Job: Job job_1481262207801_0093 failed with state FAILED due to: Application application_1481262207801_0093 failed 2 times due to AM Container for appattempt_1481262207801_0093_000002 exited with exitCode: -1000
For more detailed output, check application tracking page:http://CDHCLUSTERn03.COMPANY.COM:8088/proxy/application_1481262207801_0093/Then, click on links to logs of each attempt.
Diagnostics: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
at org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:491)
at org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:771)
at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$3.call(LoadBalancingKMSClientProvider.java:185)
at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$3.call(LoadBalancingKMSClientProvider.java:181)
at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:94)
at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:181)
at org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:388)
at org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1420)
at org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:1490)
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:311)
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:305)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:305)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:778)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:367)
at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:265)
at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:61)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:357)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:356)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:60)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:306)
at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:196)
at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:127)
at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:215)
at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:322)
at org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:485)
at org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:480)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
at org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:480)
... 29 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:285)
at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:261)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:261)
... 39 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:306)
at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:196)
at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:127)
at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:215)
at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:322)
at org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:485)
at org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:480)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
at org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:480)
at org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:771)
at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$3.call(LoadBalancingKMSClientProvider.java:185)
at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$3.call(LoadBalancingKMSClientProvider.java:181)
at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:94)
at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:181)
at org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:388)
at org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1420)
at org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:1490)
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:311)
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:305)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:305)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:778)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:367)
at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:265)
at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:61)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:357)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:356)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:60)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:285)
at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:261)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:261)
... 39 more
Caused by: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:285)
at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:261)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:261)
at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:196)
at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:127)
at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:215)
at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:322)
at org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:485)
at org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:480)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
at org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:480)
at org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:771)
at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$3.call(LoadBalancingKMSClientProvider.java:185)
at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$3.call(LoadBalancingKMSClientProvider.java:181)
at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:94)
at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:181)
at org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:388)
at org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1420)
at org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:1490)
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:311)
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:305)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:305)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:778)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:367)
at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:265)
at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:61)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:357)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:356)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:60)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Failing this attempt. Failing the application.
16/12/18 22:10:05 INFO mapreduce.Job: Counters: 0
Job Finished in 6.231 seconds
java.io.FileNotFoundException: File does not exist: hdfs://CDHCLUSTER-ns/user/sampleuser/QuasiMonteCarlo_1482120585946_2008146201/out/reduce-out
at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1219)
at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1211)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1211)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1817)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1841)
at org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:314)
at org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:354)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:363)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

[#2]

Hi,

I don't know if the map/reduce job you are submitting is Kerberos compatible. That is the first check to do.

Then, if that job is Kerberos compatible, it might need some settings like supplying a jaas configuration.

The kinit of a ticket is sometimes not enough.

For example, when running the map/reduce job "MapReduceIndexerTool", you need to supply a jaas configuration.

HADOOP_OPTS="-Djava.security.auth.login.config=/home/user/jaas.conf" \
hadoop jar MapReduceIndexerTool

See: https://www.cloudera.com/documentation/enterprise/5-4-x/topics/cdh_sg_search_security.html

[#3]

The job I'm using for testing is a standard Hadoop Pi number calculation job that comes with CDH and doesn't require any additional settings per my knowledge. This job is used as per CDH documentation for cluster testing - please see Step 17: Verify that Kerberos Security is Working for details. Here is excerpt - I'm using parcel-based CDH setup:

To verify that Kerberos security is working:
Acquire Kerberos credentials for your user account.
$ kinit USERNAME@YOUR-LOCAL-REALM.COM
Enter a password when prompted.
Submit a sample pi calculation as a test MapReduce job.

....

If you have a parcel-based setup, use the following command instead:

$ hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop-0.20-mapreduce/hadoop-examples.jar pi 10 10000
Number of Maps = 10
Samples per Map = 10000
...
Job Finished in 30.958 seconds
Estimated value of Pi is 3.14120000000000000000

[#4]

You are right, I just tested it and there is no need for additional settings (other than having initialized the kerberos ticket).

From what I read on your first post, it seems the same job do run successfully for users which do not have their home folder in HDFS encrypted ? (for the same kerberos realm) ?

If that is the case, I would open a SR ticket in your shoe. It would be the quickest way for obtaining a feedback from Cloudera on the matter (if there is an incompatiblity or some particular settings for this particular use case).

[#5]

That is correct, users with un-encrypted HDFS homes are not having this issue running this test job from edge node. As soon as user gets HDFS home folder encrypted (HDFS encryption zone created for respective /user/username folder) - this issue starts affecting that user and standadrd test described will fail for it when running from edge node. Also, as I mentioned in my initial post, this test is successfull regardless of user HDFS home encryption status when executed from actual cluster node (vs. edge nodes). 

[#6]

Hi,

If it's not working only on edge nodes then there might be some configuration issue leading to that.

What difference do you make beetwen "cluster nodes" and "edge nodes" ?

Meaning : what roles are distributed to your edge nodes ?

- For example, did you assign the HDFS&Yarn "gateway" role to your edge nodes ?

- If no, try doing it

- If yes, try redeploying the client configuration

Might be something else.

[#7]

Edge nodes are managed by CM and have the following roles on them:

• HDFS Gateway
• HDFS NFS Gateway
• Hive Gateway
• Spark Gateway
• YARN (MR2 Included) Gateway

Latest client configurations are deployed on them via CM and re-startable roles have been restarted.

[#8 · Accepted solution]

I was able to resolve the issue. It occurs because of the public-only network access from the client (edge node) to a multi-homed cluster environment (Oracle Big Data appliance in my case) and is also related to the bug MAPREDUCE-6484 . Patch is available for it and in my case it was already included in CDH 5.7.1 (CDH 5.7.1 Release Notes). However, there was an additional setting that needed to be done on Yarn to make it work:

1. Token service naming behavior needed to be changed via core-site.xml. Under CM > YARN > Configuration > Scope: YARN (Service-Wide) > Category: Advanced > "YARN Service Advanced Configuration Snippet (Safety Valve) for core-site.xml" add below property:
  

<property>
 <name>hadoop.security.token.service.use_ip</name>
 <value>false</value>
 </property>

2. Save the configuration change.
3. Deploy Client Configurations for YARN. Restart YARN Services as needed.

The details on above setting and the discussion can be found at HADOOP-7510