Hbase ExportSnapshot between clusters on different realms/KDCs ?

NVidak · ‎05-13-2019

Hello,

I'm trying to run ExportSnapshot between two clusters on different realms/KDCs. Is this supported?

I'm getting:

Java HotSpot(TM) 64-Bit Server VM warning: Using incremental CMS is deprecated and will likely be removed in a future release
19/05/14 08:35:56 WARN security.UserGroupInformation: PriviledgedActionException as:hdfs (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
19/05/14 08:35:56 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
19/05/14 08:35:56 WARN security.UserGroupInformation: PriviledgedActionException as:hdfs (auth:KERBEROS) cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
Exception in thread "main" java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "bda1node04.zbo/192.168.10.53"; destination host is: "bda1node02.zbo":8020; 
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
        at org.apache.hadoop.ipc.Client.call(Client.java:1508)
        at org.apache.hadoop.ipc.Client.call(Client.java:1441)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
        at com.sun.proxy.$Proxy11.getFileInfo(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:786)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:258)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
        at com.sun.proxy.$Proxy12.getFileInfo(Unknown Source)
        at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2167)
        at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1265)
        at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1261)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1261)
        at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1418)
        at org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:959)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:1094)
        at org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:1098)
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
        at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:718)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
        at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:681)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:769)
        at org.apache.hadoop.ipc.Client$Connection.access$3000(Client.java:396)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1557)
        at org.apache.hadoop.ipc.Client.call(Client.java:1480)
        ... 21 more

hadoop distcp works between those two clusters but only with

-Dmapreduce.job.hdfs-servers.token-renewal.exclude

option.

Harsh J · ‎05-14-2019

Yes it is supported in the same manner as with DistCp (the same exclude config needs to be used if you have a realm trust situation matching YARN-3021 [1]).

Your error however indicates that you do not have a valid Kerberos TGT on the host you're running the command on when it tries to communicate with the destination HDFS as part of job prep [2]. This is well before any MR or token work comes into play.

Are you able to perform a 'hdfs dfs -ls hdfs://remote-fs/any/path' command successfully after your kinit from the same shell you're using to submit the Export Snapshot job?

[1] - https://issues.apache.org/jira/browse/YARN-3021
[2] - https://github.infra.cloudera.com/CDH/hbase/blob/cdh5.15.2-release/hbase-server/src/main/java/org/ap...

NVidak · ‎05-15-2019

Yes. I'm able to list the target hdfs from the same shell (bda1node04 node). but the error is still here .....

Java HotSpot(TM) 64-Bit Server VM warning: Using incremental CMS is deprecated and will likely be removed in a future release
19/05/15 08:49:35 INFO snapshot.ExportSnapshot: Copy Snapshot Manifest
19/05/15 08:49:36 INFO hdfs.DFSClient: Created token for hdfs: HDFS_DELEGATION_TOKEN owner=hdfs@ZABA.ZBO, renewer=yarn, realUser=, issueDate=1557902976472, maxDate=1558507776472, sequenceNumber=20031, masterKeyId=997 on ha-hdfs:bdaprod-ns
19/05/15 08:49:36 INFO security.TokenCache: Got dt for hdfs://bdaprod-ns; Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:bdaprod-ns, Ident: (token for hdfs: HDFS_DELEGATION_TOKEN owner=hdfs@ZABA.ZBO, renewer=yarn, realUser=, issueDate=1557902976472, maxDate
=1558507776472, sequenceNumber=20031, masterKeyId=997)
19/05/15 08:49:36 INFO hdfs.DFSClient: Created token for hdfs: HDFS_DELEGATION_TOKEN owner=hdfs@ZABA.ZBO, renewer=, realUser=, issueDate=1557902976563, maxDate=1558507776563, sequenceNumber=34300, masterKeyId=1145 on 192.168.10.51:8020
19/05/15 08:49:36 INFO security.TokenCache: Got dt for hdfs://bda1node02.zbo; Kind: HDFS_DELEGATION_TOKEN, Service: 192.168.10.51:8020, Ident: (token for hdfs: HDFS_DELEGATION_TOKEN owner=hdfs@ZABA.ZBO, renewer=, realUser=, issueDate=1557902976563, maxDate
=1558507776563, sequenceNumber=34300, masterKeyId=1145)
19/05/15 08:49:37 INFO snapshot.ExportSnapshot: Loading Snapshot 'csrawSnapshot' hfile list
19/05/15 08:49:37 INFO Configuration.deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available
19/05/15 08:49:37 INFO mapreduce.JobSubmitter: number of splits:3
19/05/15 08:49:38 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1557480197033_0132
19/05/15 08:49:38 INFO mapreduce.JobSubmitter: Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:bdaprod-ns, Ident: (token for hdfs: HDFS_DELEGATION_TOKEN owner=hdfs@ZABA.ZBO, renewer=yarn, realUser=, issueDate=1557902976472, maxDate=1558507776472, sequenceNum
ber=20031, masterKeyId=997)
19/05/15 08:49:38 INFO mapreduce.JobSubmitter: Kind: HDFS_DELEGATION_TOKEN, Service: 192.168.10.51:8020, Ident: (token for hdfs: HDFS_DELEGATION_TOKEN owner=hdfs@ZABA.ZBO, renewer=, realUser=, issueDate=1557902976563, maxDate=1558507776563, sequenceNumber=
34300, masterKeyId=1145)
19/05/15 08:49:38 INFO impl.YarnClientImpl: Submitted application application_1557480197033_0132
19/05/15 08:49:38 INFO mapreduce.Job: The url to track the job: http://bda1node04.zbo:8088/proxy/application_1557480197033_0132/
19/05/15 08:49:38 INFO mapreduce.Job: Running job: job_1557480197033_0132
19/05/15 08:49:47 INFO mapreduce.Job: Job job_1557480197033_0132 running in uber mode : false
19/05/15 08:49:47 INFO mapreduce.Job:  map 0% reduce 0%
19/05/15 08:49:54 INFO mapreduce.Job: Task Id : attempt_1557480197033_0132_m_000000_0, Status : FAILED
Error: java.io.IOException: Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: "bda1node05.zbo/192.168.10.54"; destination host 
is: "bda1node02.zbo":8020; 
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
        at org.apache.hadoop.ipc.Client.call(Client.java:1508)
        at org.apache.hadoop.ipc.Client.call(Client.java:1441)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
        at com.sun.proxy.$Proxy14.getFileInfo(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:786)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:258)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
        at com.sun.proxy.$Proxy15.getFileInfo(Unknown Source)
        at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2167)
        at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1265)
        at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1261)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1261)
        at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1418)
        at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.copyFile(ExportSnapshot.java:261)
        at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.map(ExportSnapshot.java:212)
        at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.map(ExportSnapshot.java:130)
        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
        at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:718)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
        at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:681)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:769)
        at org.apache.hadoop.ipc.Client$Connection.access$3000(Client.java:396)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1557)
        at org.apache.hadoop.ipc.Client.call(Client.java:1480)
        ... 28 more
Caused by: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
        at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:172)
        at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:396)

NVidak · ‎05-15-2019

I'm also able to successfully run hadoop distcp between the same source and target