Also if I run distcp command on this getting snapshot not found error simply .. providing the details of different path .. 2018-07-14 08:14:30 INFO Initiating replication for the directory: /sas/data/prod/ifrs9/data Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0 18/07/14 08:14:34 WARN tools.OptionsParser: -delete and -diff are mutually exclusive. The -delete option will be ignored. 18/07/14 08:14:35 INFO tools.DistCp: Input Options: DistCpOptions{atomicCommit=false, syncFolder=true, deleteMissing=false, ignoreFailures=true, overwrite=false, skipCRC=true, blocking=true, numListstatusThreads=0, maxMaps=100, mapBandwidth=100, sslConfigurationFile='null', copyStrategy='uniformsize', preserveStatus=[REPLICATION, BLOCKSIZE, USER, GROUP, PERMISSION], preserveRawXattrs=false, atomicWorkPath=null, logPath=null, sourceFileListing=null, sourcePaths=[hdfs://HDP05/sas/data/prod/ifrs9/data], targetPath=hdfs://HDP39/sas/data/prod/ifrs9/data, targetPathExists=true, filtersFile='null'} 18/07/14 08:14:36 INFO impl.TimelineClientImpl: Timeline service address: http://<<RMhost>>:8188/ws/v1/timeline/ 18/07/14 08:14:36 WARN retry.RetryInvocationHandler: Exception while invoking class org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getSnapshotDiffReport over <<HDP39NNhost>>/1<<HDP39 NN>>:8020. Not retrying because try once and fail. org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.SnapshotException): Cannot find the snapshot of directory /sas/data/prod/ifrs9/data with name snapshot_sasdataprodifrs9data_201803021147 at org.apache.hadoop.hdfs.server.namenode.snapshot.DirectorySnapshottableFeature.getSnapshotByName(DirectorySnapshottableFeature.java:285) at org.apache.hadoop.hdfs.server.namenode.snapshot.DirectorySnapshottableFeature.computeDiff(DirectorySnapshottableFeature.java:257) at org.apache.hadoop.hdfs.server.namenode.snapshot.SnapshotManager.diff(SnapshotManager.java:372) at org.apache.hadoop.hdfs.server.namenode.FSDirSnapshotOp.getSnapshotDiffReport(FSDirSnapshotOp.java:155) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getSnapshotDiffReport(FSNamesystem.java:7688) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getSnapshotDiffReport(NameNodeRpcServer.java:1792) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getSnapshotDiffReport(ClientNamenodeProtocolServerSideTranslatorPB.java:1149) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2273) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2269) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2267) at org.apache.hadoop.ipc.Client.call(Client.java:1455) at org.apache.hadoop.ipc.Client.call(Client.java:1392) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy10.getSnapshotDiffReport(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getSnapshotDiffReport(ClientNamenodeProtocolTranslatorPB.java:1107) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:258) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104) at com.sun.proxy.$Proxy11.getSnapshotDiffReport(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.getSnapshotDiffReport(DFSClient.java:2791) at org.apache.hadoop.hdfs.DistributedFileSystem$38.doCall(DistributedFileSystem.java:1816) at org.apache.hadoop.hdfs.DistributedFileSystem$38.doCall(DistributedFileSystem.java:1812) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getSnapshotDiffReport(DistributedFileSystem.java:1812) at org.apache.hadoop.tools.DistCpSync.checkNoChange(DistCpSync.java:252) at org.apache.hadoop.tools.DistCpSync.preSyncCheck(DistCpSync.java:92) at org.apache.hadoop.tools.DistCpSync.sync(DistCpSync.java:124) at org.apache.hadoop.tools.DistCp.createAndSubmitJob(DistCp.java:179) at org.apache.hadoop.tools.DistCp.execute(DistCp.java:154) at org.apache.hadoop.tools.DistCp.run(DistCp.java:127) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.tools.DistCp.main(DistCp.java:461) 18/07/14 08:14:36 WARN tools.DistCp: Failed to compute snapshot diff on hdfs://HDP39/sas/data/prod/ifrs9/data org.apache.hadoop.hdfs.protocol.Snapshot Exception: Cannot find the snapshot of directory /sas/data/prod/ifrs9/data with name snapshot_sasdataprodifrs9data_201803021147 at org.apache.hadoop.hdfs.server.namenode.snapshot.DirectorySnapshottableFeature.getSnapshotByName(DirectorySnapshottableFeature.java:285) at org.apache.hadoop.hdfs.server.namenode.snapshot.DirectorySnapshottableFeature.computeDiff(DirectorySnapshottableFeature.java:257) at org.apache.hadoop.hdfs.server.namenode.snapshot.SnapshotManager.diff(SnapshotManager.java:372) at org.apache.hadoop.hdfs.server.namenode.FSDirSnapshotOp.getSnapshotDiffReport(FSDirSnapshotOp.java:155) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getSnapshotDiffReport(FSNamesystem.java:7688) ... View more

Sure I am able to run the HDFS commands of PROD(HDP05) like list from DR(HDP39) cluster .. but unable to run snapshotdiff which is failing.. here is the stack trace.. user hdp05-drmin is a super user on both the clusters Below the snapshot command running [hdp05-drmin ~]$ hdfs snapshotDiff hdfs://HDP05/data/IFRS9/prod snapshot_dataIFRS9prod_201807140834 snapshot_dataIFRS9prod_201807140821 Exception in thread "main" java.lang.IllegalArgumentException: Wrong FS: hdfs://HDP05/data/IFRS9/prod, expected: hdfs://HDP39 at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:651) at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:196) at org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:108) at org.apache.hadoop.hdfs.DistributedFileSystem$38.doCall(DistributedFileSystem.java:1816) at org.apache.hadoop.hdfs.DistributedFileSystem$38.doCall(DistributedFileSystem.java:1812) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getSnapshotDiffReport(DistributedFileSystem.java:1812) at org.apache.hadoop.hdfs.tools.snapshot.SnapshotDiff.run(SnapshotDiff.java:88) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90) at org.apache.hadoop.hdfs.tools.snapshot.SnapshotDiff.main(SnapshotDiff.java:100) Below same list command working fine.. [hdp05-drmin ~]$ hdfs dfs -ls hdfs://HDP05/data/IFRS9/prod Found 9 items drwxr-xr-x - HDP05-ss-uk HDP05-ss-uk-group 0 2018-07-13 13:03 hdfs://HDP05/data/IFRS9/prod/DisAgg drwxr-xr-x - HDP05-ss-uk HDP05-ss-uk-group 0 2017-10-11 13:06 hdfs://HDP05/data/IFRS9/prod/IFRS9 drwxr-xr-x - HDP05-ss-uk HDP05-ss-uk-group 0 2018-07-03 15:25 hdfs://HDP05/data/IFRS9/prod/LIVE ... View more

@Jitendra Yadav can you have some insights on this .. ... View more

There is challenge I am facing .. when I am running the snapshotdiff from a remote cluster it is failing with snapshot not found error even though it is available .. do we have any solution for this .. we built a DR cluster and running distcp from DR to utilize the DR resources instead of overloading the PROD .. any solution how this can be achived.. ... View more

both the clusters are kerberised ... View more

Even I am facing this issue ..where I am having multiple clusters for other clusters I am not getting this error only for one cluster I am getting the error..I checked Java version on the working and non-working clusters both are having same version .. ... View more

It looks like you followed the kerberos instructions to install the Hive view ..do recheck once ..on view you no need to give any security related settings ..if your cluster is not enabled for kerberoes .. ... View more

Thanks for the info I will try this today and update you how it worked.....by the way do we have any yum repo file from where I can get the files.. ... View more

here is the error and setting .. ... View more

With in Hive session ... View more

Jitendra...when you say balance_switch true ...what it is doing ...the region servers data was already balanced. Here my issue us the 56-regions for the given table are loaded only into two region server instead of eight servers ... View more

we are using HDP2.2 and Hbase 0.98.and Hive 0.14 yes all the region servers are already balanced ..but for this table all 56 regions are stored only in two servers..for other tables I see regions are distributed between all region servers.. ... View more

thanks for the time ..but I tried the min table size as noconditionaltask.size still it took it is own value and failed with mapjoinmemory exhausted error..for that reason I am suspecting is there any max limit.. ... View more