Created on 05-13-2019 11:52 PM - edited 09-16-2022 07:23 AM
Hello,
I'm trying to run ExportSnapshot between two clusters on different realms/KDCs. Is this supported?
I'm getting:
Java HotSpot(TM) 64-Bit Server VM warning: Using incremental CMS is deprecated and will likely be removed in a future release 19/05/14 08:35:56 WARN security.UserGroupInformation: PriviledgedActionException as:hdfs (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] 19/05/14 08:35:56 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] 19/05/14 08:35:56 WARN security.UserGroupInformation: PriviledgedActionException as:hdfs (auth:KERBEROS) cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] Exception in thread "main" java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "bda1node04.zbo/192.168.10.53"; destination host is: "bda1node02.zbo":8020; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1441) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230) at com.sun.proxy.$Proxy11.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:786) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:258) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104) at com.sun.proxy.$Proxy12.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2167) at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1265) at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1261) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1261) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1418) at org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:959) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:1094) at org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:1098) Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:718) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:681) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:769) at org.apache.hadoop.ipc.Client$Connection.access$3000(Client.java:396) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1557) at org.apache.hadoop.ipc.Client.call(Client.java:1480) ... 21 more
hadoop distcp works between those two clusters but only with
-Dmapreduce.job.hdfs-servers.token-renewal.exclude
option.
Created 05-14-2019 07:45 PM
Created 05-15-2019 12:41 AM
Yes. I'm able to list the target hdfs from the same shell (bda1node04 node). but the error is still here .....
Java HotSpot(TM) 64-Bit Server VM warning: Using incremental CMS is deprecated and will likely be removed in a future release 19/05/15 08:49:35 INFO snapshot.ExportSnapshot: Copy Snapshot Manifest 19/05/15 08:49:36 INFO hdfs.DFSClient: Created token for hdfs: HDFS_DELEGATION_TOKEN owner=hdfs@ZABA.ZBO, renewer=yarn, realUser=, issueDate=1557902976472, maxDate=1558507776472, sequenceNumber=20031, masterKeyId=997 on ha-hdfs:bdaprod-ns 19/05/15 08:49:36 INFO security.TokenCache: Got dt for hdfs://bdaprod-ns; Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:bdaprod-ns, Ident: (token for hdfs: HDFS_DELEGATION_TOKEN owner=hdfs@ZABA.ZBO, renewer=yarn, realUser=, issueDate=1557902976472, maxDate =1558507776472, sequenceNumber=20031, masterKeyId=997) 19/05/15 08:49:36 INFO hdfs.DFSClient: Created token for hdfs: HDFS_DELEGATION_TOKEN owner=hdfs@ZABA.ZBO, renewer=, realUser=, issueDate=1557902976563, maxDate=1558507776563, sequenceNumber=34300, masterKeyId=1145 on 192.168.10.51:8020 19/05/15 08:49:36 INFO security.TokenCache: Got dt for hdfs://bda1node02.zbo; Kind: HDFS_DELEGATION_TOKEN, Service: 192.168.10.51:8020, Ident: (token for hdfs: HDFS_DELEGATION_TOKEN owner=hdfs@ZABA.ZBO, renewer=, realUser=, issueDate=1557902976563, maxDate =1558507776563, sequenceNumber=34300, masterKeyId=1145) 19/05/15 08:49:37 INFO snapshot.ExportSnapshot: Loading Snapshot 'csrawSnapshot' hfile list 19/05/15 08:49:37 INFO Configuration.deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available 19/05/15 08:49:37 INFO mapreduce.JobSubmitter: number of splits:3 19/05/15 08:49:38 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1557480197033_0132 19/05/15 08:49:38 INFO mapreduce.JobSubmitter: Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:bdaprod-ns, Ident: (token for hdfs: HDFS_DELEGATION_TOKEN owner=hdfs@ZABA.ZBO, renewer=yarn, realUser=, issueDate=1557902976472, maxDate=1558507776472, sequenceNum ber=20031, masterKeyId=997) 19/05/15 08:49:38 INFO mapreduce.JobSubmitter: Kind: HDFS_DELEGATION_TOKEN, Service: 192.168.10.51:8020, Ident: (token for hdfs: HDFS_DELEGATION_TOKEN owner=hdfs@ZABA.ZBO, renewer=, realUser=, issueDate=1557902976563, maxDate=1558507776563, sequenceNumber= 34300, masterKeyId=1145) 19/05/15 08:49:38 INFO impl.YarnClientImpl: Submitted application application_1557480197033_0132 19/05/15 08:49:38 INFO mapreduce.Job: The url to track the job: http://bda1node04.zbo:8088/proxy/application_1557480197033_0132/ 19/05/15 08:49:38 INFO mapreduce.Job: Running job: job_1557480197033_0132 19/05/15 08:49:47 INFO mapreduce.Job: Job job_1557480197033_0132 running in uber mode : false 19/05/15 08:49:47 INFO mapreduce.Job: map 0% reduce 0% 19/05/15 08:49:54 INFO mapreduce.Job: Task Id : attempt_1557480197033_0132_m_000000_0, Status : FAILED Error: java.io.IOException: Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: "bda1node05.zbo/192.168.10.54"; destination host is: "bda1node02.zbo":8020; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1441) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230) at com.sun.proxy.$Proxy14.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:786) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:258) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104) at com.sun.proxy.$Proxy15.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2167) at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1265) at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1261) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1261) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1418) at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.copyFile(ExportSnapshot.java:261) at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.map(ExportSnapshot.java:212) at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.map(ExportSnapshot.java:130) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:718) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:681) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:769) at org.apache.hadoop.ipc.Client$Connection.access$3000(Client.java:396) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1557) at org.apache.hadoop.ipc.Client.call(Client.java:1480) ... 28 more Caused by: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:172) at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:396)
Created 05-15-2019 03:22 AM
I'm also able to successfully run hadoop distcp between the same source and target