Hello Team,   Can anyone please help me with your comments.   Thanks, Vinod ... View more

Hello @Shelton    I have added above property in yarn-site.xml for nodemanager and restarted, Still i see same issue and same logs in nodemanager and it is in unknown state,   WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: The Auxilurary Service named 'mapreduce_shuffle' in the configuration is for class class org.apache.hadoop.mapred.ShuffleHandler which has a name of 'httpshuffle'. Because these are not the same tools trying to send ServiceData and read Service Meta Data may have issues unless the refer to the name in the config   Please give me your valuable response.   Thanks, Vinod ... View more

Hello @Shelton    Any suggestions ? My only doubt is, As other nodemanagers are running fine in the cluster why these newly added nodemanagers were going to unknown state?   Regards, Vinod ... View more

Hello @Shelton    Yes we are using YARN and other Nodemanagers are up and running with same configurations but for newly added nodes we are facing above logs and they are going to unknown state.   Yes i verified the above parameter but i dont see the above parameter but in yarn configurations page, I can see " Enable Shuffle Auxiliary Service " was enabled.   Please suggest me your comments.   Thanks, Vinod ... View more

Hello @rblough    First of all thank you so much for your response..!!   1. The X and Y environments are having different HBase versions. 2. HBase X version is 1.3.1 and HBase Y Version is 1.0.0 3. We are not clearing the table but following below steps,   echo "disable '$1'" | hbase shell echo "restore_snapshot '$1_SNAPSHOT_$DATE'" | hbase shell echo "enable '$1'" | hbase shell   Daily we are doing the same but sometimes we are seeing the counts mismatch. So, we are doing truncating and restoring the same snapshots and working fine in Y environment.   Please help us your inputs.   Thanks & Regards, Vinod   ... View more

Hello Team,   Can someone please help me with your inputs.   Best Regards, Vinod ... View more

Hi @Shelton    Could you please help me.   Thanks, Vinod ... View more

Hello Team,   Any help would be appreciated.   Best Regards, Vinod     ... View more

Hello @Shelton    Thanks for your response, I have just installed CDH 6.0.1 with 6 nodes and in another server i have installed docker in edge node.   I dont have any reference to integrate docker with yarn, If you can help me with any steps to integrate docker that would be great.   Thanks, Vinod ... View more

Hello Team,   Can someone please give me your response and help me with this issue.   Regards, Vinod ... View more

Hi @Shelton ,   Thanks for your response and yes i have tried re-genrating key-tab files, But no luck. And the above two servers are master nodes (Zookeeper, journal nodes and other master services are running).   Please let me know if you need any other details.   Thanks, Vinod   ... View more

Hello @Shelton @venkatsambath    Hope you all doing good. None of the configuration files are bundled with our jars and they use our latest updated config files in the cluster only. What i am missing here ? what could be the reason for failing after enabling Kerberos ?   Please do the needful.   Best Regards, Vinod   ... View more

Hey @satz    There is no other process is running at that time and only one job is running and frequently it is failing with below error.   Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /user/hbase/.hbase-snapshot/.tmp/   And the import job is failing. But when i re-ran it is completing with out any bugs.   Please help me with your thoughts.   Thanks, Vinod   ... View more

Hello @Shelton,   There is no logs generated on that time interval. But i will check for yourquestions with my team and get back to you. Thanks for responding.   Best Regards, Vinod ... View more

Hello @venkatsambath @Shelton    Can some one please revert back and that would be great for me.   Best Regards, Vinod       ... View more

Hi @venkatsambath ,   Find below one of our application logs ,     INFO 2020Mar27 01:13:07,422 main com.class.engineering.portfolio.finalresolution.main.MrFinalResolver: Sleeping for >> 300000 ms ERROR 2020Mar27 01:18:16,485 main com.class.engineering.portfolio.finalresolution.main.MrFinalResolver: Exception occurred while checking for isReadyToRun flag >> Can't get the location org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:308) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:149) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:57) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:293) at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:268) at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:140) at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:135) at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:888) at com.class.engineering.portfolio.creditinghadoop.config.ConfigTbl.readValue(ConfigTbl.java:115) at com.class.engineering.portfolio.creditinghadoop.config.ConfigDao.read(ConfigDao.java:63) at com.class.engineering.portfolio.finalresolution.main.MrFinalResolver.isReadyToRun(MrFinalResolver.java:344) at com.class.engineering.portfolio.finalresolution.main.MrFinalResolver.main(MrFinalResolver.java:116) Caused by: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to hostname003.enterprisenet.org/1.1.1.1:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to hostname003.enterprisenet.org/1.1.1.1:60020 is closing. Call id=4045, waitTime=2 at org.apache.hadoop.hbase.ipc.RpcClientImpl.wrapException(RpcClientImpl.java:1243) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1214) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:216) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:300) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.get(ClientProtos.java:31865) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRowOrBefore(ProtobufUtil.java:1580) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1294) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1126) at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:299) ... 12 more Caused by: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to hostname003.enterprisenet.org/1.1.1.1:60020 is closing. Call id=4045, waitTime=2 at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.cleanupCalls(RpcClientImpl.java:1033) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.close(RpcClientImpl.java:840) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.run(RpcClientImpl.java:568) ERROR 2020Mar27 01:18:25,554 main com.class.engineering.portfolio.resolution.util.HLogUtil: Exception occured while eriting log message >> Failed 1 action: IOException: 1 time, org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 action: IOException: 1 time, at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:227) at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1700(AsyncProcess.java:207) at org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1658) at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:208) at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:183) at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1496) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1107) at com.class.engineering.portfolio.creditinghadoop.log.HLogTbl.write(HLogTbl.java:102) at com.class.engineering.portfolio.creditinghadoop.log.HLogDao.write(HLogDao.java:50) at com.class.engineering.portfolio.creditinghadoop.log.HLog.writeMsg(HLog.java:20) at com.class.engineering.portfolio.resolution.util.HLogUtil.writeMsg(HLogUtil.java:18) at com.class.engineering.portfolio.finalresolution.main.MrFinalResolver.isReadyToRun(MrFinalResolver.java:358) at com.class.engineering.portfolio.finalresolution.main.MrFinalResolver.main(MrFinalResolver.java:116) INFO 2020Mar27 01:18:25,555 main com.class.engineering.portfolio.finalresolution.main.MrFinalResolver: isReady false. Elapsed time: 18130 ms. INFO 2020Mar27 01:18:25,555 main com.class.engineering.portfolio.finalresolution.main.MrFinalResolver: Sleeping for >> 300000 ms   Please revert back.   Best Regards, Vinod ... View more

Hi @venkatsambath    One problem is resolved, I mean to say i am able to run sample yarn job.   Found that one of server having some permission issue and deleted /disk(1,2,3,4,5)/yarn/nm directory and restarted the nodemanager and ran yarn job. Its worked.   And i tried our application related job and this time getting some different error's,  This job i am running in hostname001 and getting below error and seeing some connection failure to hostname003 server. ERROR 2020Mar27 02:35:46,850 main com.class.engineering.portfolio.dmxsloader.main.DMXSLoaderMain: org.apache.hadoop.hbase.client.RetriesExhaustedException thrown: Can't get the location org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:308) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:149) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:57) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:293) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:268) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:140) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:135) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:888) ~[DMXSLoader-0.0.31.jar:0.0.31] at com.class.engineering.portfolio.dmxsloader.main.DMXSLoaderMain.hasStagingData(DMXSLoaderMain.java:304) [DMXSLoader-0.0.31.jar:0.0.31] at com.class.engineering.portfolio.dmxsloader.main.DMXSLoaderMain.main(DMXSLoaderMain.java:375) [DMXSLoader-0.0.31.jar:0.0.31] Caused by: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to hostname003.enterprisenet.org/10.7.54.13:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to hostname003.enterprisenet.org/10.7.54.13:60020 is closing. Call id=2, waitTime=12     Regards, Vinod   ... View more