Hello @Shelton,   There is no logs generated on that time interval. But i will check for yourquestions with my team and get back to you. Thanks for responding.   Best Regards, Vinod ... View more

Hello @venkatsambath @Shelton    Can some one please revert back and that would be great for me.   Best Regards, Vinod       ... View more

Hi @venkatsambath ,   Find below one of our application logs ,     INFO 2020Mar27 01:13:07,422 main com.class.engineering.portfolio.finalresolution.main.MrFinalResolver: Sleeping for >> 300000 ms ERROR 2020Mar27 01:18:16,485 main com.class.engineering.portfolio.finalresolution.main.MrFinalResolver: Exception occurred while checking for isReadyToRun flag >> Can't get the location org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:308) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:149) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:57) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:293) at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:268) at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:140) at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:135) at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:888) at com.class.engineering.portfolio.creditinghadoop.config.ConfigTbl.readValue(ConfigTbl.java:115) at com.class.engineering.portfolio.creditinghadoop.config.ConfigDao.read(ConfigDao.java:63) at com.class.engineering.portfolio.finalresolution.main.MrFinalResolver.isReadyToRun(MrFinalResolver.java:344) at com.class.engineering.portfolio.finalresolution.main.MrFinalResolver.main(MrFinalResolver.java:116) Caused by: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to hostname003.enterprisenet.org/1.1.1.1:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to hostname003.enterprisenet.org/1.1.1.1:60020 is closing. Call id=4045, waitTime=2 at org.apache.hadoop.hbase.ipc.RpcClientImpl.wrapException(RpcClientImpl.java:1243) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1214) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:216) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:300) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.get(ClientProtos.java:31865) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRowOrBefore(ProtobufUtil.java:1580) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1294) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1126) at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:299) ... 12 more Caused by: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to hostname003.enterprisenet.org/1.1.1.1:60020 is closing. Call id=4045, waitTime=2 at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.cleanupCalls(RpcClientImpl.java:1033) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.close(RpcClientImpl.java:840) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.run(RpcClientImpl.java:568) ERROR 2020Mar27 01:18:25,554 main com.class.engineering.portfolio.resolution.util.HLogUtil: Exception occured while eriting log message >> Failed 1 action: IOException: 1 time, org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 action: IOException: 1 time, at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:227) at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1700(AsyncProcess.java:207) at org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1658) at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:208) at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:183) at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1496) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1107) at com.class.engineering.portfolio.creditinghadoop.log.HLogTbl.write(HLogTbl.java:102) at com.class.engineering.portfolio.creditinghadoop.log.HLogDao.write(HLogDao.java:50) at com.class.engineering.portfolio.creditinghadoop.log.HLog.writeMsg(HLog.java:20) at com.class.engineering.portfolio.resolution.util.HLogUtil.writeMsg(HLogUtil.java:18) at com.class.engineering.portfolio.finalresolution.main.MrFinalResolver.isReadyToRun(MrFinalResolver.java:358) at com.class.engineering.portfolio.finalresolution.main.MrFinalResolver.main(MrFinalResolver.java:116) INFO 2020Mar27 01:18:25,555 main com.class.engineering.portfolio.finalresolution.main.MrFinalResolver: isReady false. Elapsed time: 18130 ms. INFO 2020Mar27 01:18:25,555 main com.class.engineering.portfolio.finalresolution.main.MrFinalResolver: Sleeping for >> 300000 ms   Please revert back.   Best Regards, Vinod ... View more

Hi @venkatsambath    One problem is resolved, I mean to say i am able to run sample yarn job.   Found that one of server having some permission issue and deleted /disk(1,2,3,4,5)/yarn/nm directory and restarted the nodemanager and ran yarn job. Its worked.   And i tried our application related job and this time getting some different error's,  This job i am running in hostname001 and getting below error and seeing some connection failure to hostname003 server. ERROR 2020Mar27 02:35:46,850 main com.class.engineering.portfolio.dmxsloader.main.DMXSLoaderMain: org.apache.hadoop.hbase.client.RetriesExhaustedException thrown: Can't get the location org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:308) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:149) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:57) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:293) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:268) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:140) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:135) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:888) ~[DMXSLoader-0.0.31.jar:0.0.31] at com.class.engineering.portfolio.dmxsloader.main.DMXSLoaderMain.hasStagingData(DMXSLoaderMain.java:304) [DMXSLoader-0.0.31.jar:0.0.31] at com.class.engineering.portfolio.dmxsloader.main.DMXSLoaderMain.main(DMXSLoaderMain.java:375) [DMXSLoader-0.0.31.jar:0.0.31] Caused by: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to hostname003.enterprisenet.org/10.7.54.13:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to hostname003.enterprisenet.org/10.7.54.13:60020 is closing. Call id=2, waitTime=12     Regards, Vinod   ... View more

Hi @venkatsambath    Thanks for your response..! Yes, i have deleted upto usercache and restarted nodemanagers, We installed 10 nodemanagers and find below stats what you asked,   hostname1.enterprisenet.org sudo namei -l /disk1/yarn/nm/usercache f: /disk1/yarn/nm/usercache dr-xr-xr-x root root / drwxr-xr-x root root disk1 drwxr-xr-x root root yarn drwxr-xr-x yarn hadoop nm drwxr-xr-x yarn yarn usercache hostname002.enterprisenet.org sudo namei -l /disk1/yarn/nm/usercache f: /disk1/yarn/nm/usercache dr-xr-xr-x root root / drwxr-xr-x root root disk1 drwxr-xr-x root root yarn drwxr-xr-x yarn hadoop nm drwxr-xr-x yarn yarn usercache hostname003.enterprisenet.org sudo namei -l /disk1/yarn/nm/usercache f: /disk1/yarn/nm/usercache dr-xr-xr-x root root / drwxr-xr-x root root disk1 drwxr-xr-x root root yarn drwxr-xr-x yarn hadoop nm drwxr-xr-x yarn yarn usercache hostname55.enterprisenet.org sudo namei -l /disk1/yarn/nm/usercache f: /disk1/yarn/nm/usercache dr-xr-xr-x root root / drwxr-xr-x root root disk1 drwxr-xr-x root root yarn drwxr-xr-x yarn hadoop nm drwxr-xr-x yarn yarn usercache hostname001.enterprisenet.org sudo namei -l /disk1/yarn/nm/usercache f: /disk1/yarn/nm/usercache dr-xr-xr-x root root / drwxr-xr-x root root disk1 drwxr-xr-x root root yarn drwxr-xr-x yarn hadoop nm drwxr-xr-x yarn yarn usercache hostname003.enterprisenet.org sudo namei -l /disk1/yarn/nm/usercache f: /disk1/yarn/nm/usercache dr-xr-xr-x root root / drwxr-xr-x root root disk1 drwxr-xr-x root root yarn drwxr-xr-x yarn hadoop nm drwxr-xr-x yarn yarn usercache hostname028.enterprisenet.org sudo namei -l /disk1/yarn/nm/usercache f: /disk1/yarn/nm/usercache dr-xr-xr-x root root / drwxr-xr-x root root disk1 drwxr-xr-x root root yarn drwxr-xr-x yarn hadoop nm drwxr-xr-x yarn yarn usercache hostname029.enterprisenet.org sudo namei -l /disk1/yarn/nm/usercache f: /disk1/yarn/nm/usercache dr-xr-xr-x root root / drwxr-xr-x root root disk1 drwxr-xr-x root root yarn drwxr-xr-x yarn hadoop nm drwxr-xr-x yarn yarn usercache hostname054.enterprisenet.org sudo namei -l /disk1/yarn/nm/usercache f: /disk1/yarn/nm/usercache dr-xr-xr-x root root / drwxr-xr-x root root disk1 drwxr-xr-x root root yarn drwxr-xr-x yarn hadoop nm drwxr-xr-x yarn yarn usercache     a. Yes, the hbase gate way is available in the same server. b. Actually its not worked with hbase user. And i have granted the permissions to mcaf  through hbase shell like below,   hbase(main):004:0> grant 'mcaf', 'RWXC' , 'TABLE1' 0 row(s) in 0.6570 seconds hbase(main):005:0> user_permission 'TABLE1' User Namespace,Table,Family,Qualifier:Permission mcaf default,TABLE1,,: [Permission: actions=READ,WRITE,EXEC,CREATE] 1 row(s) in 0.3960 seconds hbase(main):006:0> grant 'mcaf', 'RWXC' , 'TABLE2' 0 row(s) in 0.5780 seconds hbase(main):007:0> user_permission 'TABLE2' User Namespace,Table,Family,Qualifier:Permission mcaf default,TABLE2,,: [Permission: actions=READ,WRITE,EXEC,CREATE] 1 row(s) in 0.4060 seconds   c. I am not find any error or warn messages in HMaster logs while i am querying.   After granting the permissions i have tested my application job and seeing same error messages like above.   Please revert back.   Best Regards, Vinod ... View more

Hi @venkatsambath @Shelton ,   As you said I deleted the cache directories and tried again. But still facing the same issue, Find below error logs and directory structures.     Error Logs: JOB: yarn jar /opt/cloudera/parcels/CDH-5.4.7-1.cdh5.4.7.p0.3/jars/hadoop-examples.jar teragen 500000000 /tmp/teragen22     20/03/26 11:36:53 INFO mapreduce.Job: Job job_1585236349590_0001 failed with state FAILED due to: Application application_1585236349590_0001 failed 2 times due to AM Container for appattempt_1585236349590_0001_000002 exited with exitCode: -1000 For more detailed output, check application tracking page:http://hostname:8088/proxy/application_1585236349590_0001/Then, click on links to logs of each attempt. Diagnostics: Application application_1585236349590_0001 initialization failed (exitCode=20) with output: main : command provided 0 main : user is mcaf main : requested yarn user is mcaf Failed to create directory /disk1/yarn/nm/usercache/mcaf - No such file or directory Failed to create directory /disk2/yarn/nm/usercache/mcaf - No such file or directory Failed to create directory /disk3/yarn/nm/usercache/mcaf - No such file or directory Failed to create directory /disk4/yarn/nm/usercache/mcaf - No such file or directory Failed to create directory /disk5/yarn/nm/usercache/mcaf - No such file or directory Failing this attempt. Failing the application. 20/03/26 11:36:53 INFO mapreduce.Job: Counters: 0     I give a try for my application job and that is also failed and below are the logs,     ERROR 2020Mar26 11:36:00,863 main com.class.engineering.portfolio.dmxsloader.main.DMXSLoaderMain: org.apache.hadoop.hbase.client.RetriesExhaustedException thrown: Can't get the location org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:308) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:149) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:57) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:293) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:268) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:140) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:135) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:888) ~[DMXSLoader-0.0.31.jar:0.0.31] at com.class.engineering.portfolio.dmxsloader.main.DMXSLoaderMain.hasStagingData(DMXSLoaderMain.java:304) [DMXSLoader-0.0.31.jar:0.0.31] at com.class.engineering.portfolio.dmxsloader.main.DMXSLoaderMain.main(DMXSLoaderMain.java:375) [DMXSLoader-0.0.31.jar:0.0.31] Caused by: java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[?:1.7.0_67] at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[?:1.7.0_67] at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[?:1.7.0_67] at sun.nio.ch.IOUtil.write(IOUtil.java:65) ~[?:1.7.0_67] at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487) ~[?:1.7.0_67] at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:63) ~[hadoop-common-2.6.0-cdh5.4.7.jar:?] at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142) ~[hadoop-common-2.6.0-cdh5.4.7.jar:?] at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:159) ~[hadoop-common-2.6.0-cdh5.4.7.jar:?] at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:117) ~[hadoop-common-2.6.0-cdh5.4.7.jar:?] at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) ~[?:1.7.0_67] at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) ~[?:1.7.0_67] at java.io.DataOutputStream.flush(DataOutputStream.java:123) ~[?:1.7.0_67] at org.apache.hadoop.hbase.ipc.IPCUtil.write(IPCUtil.java:246) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.ipc.IPCUtil.write(IPCUtil.java:234) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:895) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:850) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1184) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:216) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:300) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.get(ClientProtos.java:31865) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRowOrBefore(ProtobufUtil.java:1580) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1294) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1126) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:299) ~[DMXSLoader-0.0.31.jar:0.0.31] ... 10 more     Directory structure:   hostname]$ sudo ls -ld /disk1/yarn/nm drwxr-xr-x 4 yarn hadoop 4096 Mar 26 11:37 /disk1/yarn/nm hostname]$ sudo ls -ld /disk1/yarn/ drwxr-xr-x 3 root root 4096 Sep 28 2015 /disk1/yarn/ hostname]$ sudo ls -ld /disk1 drwxr-xr-x 9 root root 4096 Sep 28 2015 /disk1     Please revert back.   Best Regards, Vinod ... View more

Hi @Shelton @venkatsambath    Can some one please help me to fix the issue ?   Best Regards, Vinod ... View more

Hi @Shelton ,   I ran above command you shared for removing 's' permissions for the directory. And then i triggered same yarn sample job and facing same issue,   ERROR: 20/03/24 12:29:24 INFO mapreduce.Job: Task Id : attempt_1585066027398_0003_m_000000_1, Status : FAILED Application application_1585066027398_0003 initialization failed (exitCode=255) with output: main : command provided 0 main : user is mcaf main : requested yarn user is mcaf Can't create directory /disk1/yarn/nm/usercache/mcaf/appcache/application_1585066027398_0003 - Permission denied Can't create directory /disk2/yarn/nm/usercache/mcaf/appcache/application_1585066027398_0003 - Permission denied Can't create directory /disk3/yarn/nm/usercache/mcaf/appcache/application_1585066027398_0003 - Permission denied Can't create directory /disk4/yarn/nm/usercache/mcaf/appcache/application_1585066027398_0003 - Permission denied Can't create directory /disk5/yarn/nm/usercache/mcaf/appcache/application_1585066027398_0003 - Permission denied Did not create any app directories     Find below directory structure.   HOSTNAME]$ sudo ls -lrt /disk2/yarn/nm/usercache/mcaf/appcache total 0 HOSTNAME]$ sudo ls -ld /disk2/yarn/nm/usercache/mcaf/appcache drwx--x--- 2 yarn yarn 4096 Mar 4 02:22 /disk2/yarn/nm/usercache/mcaf/appcache HOSTNAME]$ sudo ls -lrt /disk2/yarn/nm/usercache/mcaf total 24 drwx--x--- 493 yarn yarn 20480 Mar 4 01:18 filecache drwx--x--- 2 yarn yarn 4096 Mar 4 02:22 appcache HOSTNAME]$ sudo ls -ld /disk2/yarn/nm/usercache/mcaf drwxr-x--- 4 yarn yarn 4096 Feb 24 01:26 /disk2/yarn/nm/usercache/mcaf HOSTNAME]$ sudo ls -ld /disk2/yarn/nm/usercache/ drwxr-xr-x 3 yarn yarn 4096 Feb 24 01:26 /disk2/yarn/nm/usercache/ HOSTNAME]$ sudo ls -lrt /disk2/yarn/nm/usercache total 4 drwxr-x--- 4 yarn yarn 4096 Feb 24 01:26 mcaf   NOTE: I have modified those permissions in all the servers.   Best Regards, Vinod ... View more

Hello @Shelton ,   Thanks for your immediate response. Find below outputs,   HOSTNAME]$ ls /disk1/yarn/nm/usercache mcaf HOSTNAME]$ ls /disk1/yarn/nm/usercache/mcaf appcache filecache HOSTNAME]$ ls -lrt /disk1/yarn/nm/usercache/mcaf total 20 drwx--x--- 397 yarn yarn 16384 Mar 4 01:18 filecache drwx--x--- 2 yarn yarn 4096 Mar 4 02:22 appcache HOSTNAME]$ ls -lrt /disk1/yarn/nm/usercache total 4 drwxr-s--- 4 mcaf yarn 4096 Feb 24 01:26 mcaf   Q1, If we enable kerberos do we needs to modify permissions to the above directory? And mcaf having sudo access.   Q2, We are using two edgenodes. Can i use the above merged.keytab in another edgenode ? Or do i needs to generate them like what i did in current edgenode ?   Best Regards, Vinod   ... View more

Hello @Shelton @venkatsambath ,   As you mention above I have done merging keytab files like, mcaf.keytab, yarn.keytab and other service keytabs. Created mcafmerged.keytab and executed using kinit -kt mcafmerged.keytab mcaf@Domain.ORG   After the above process i am able to access hdfs, hbase tables using hbase shell and able to see yarn applications -list.   But when i run below sample yarn job,   yarn jar /opt/cloudera/parcels/CDH-5.4.7-1.cdh5.4.7.p0.3/jars/hadoop-examples.jar teragen 500000000 /tmp/teragen44   Getting below error's, Can't create directory /disk1/yarn/nm/usercache/mcaf/appcache/application_1585026002165_0001 - Permission denied Can't create directory /disk2/yarn/nm/usercache/mcaf/appcache/application_1585026002165_0001 - Permission denied Can't create directory /disk3/yarn/nm/usercache/mcaf/appcache/application_1585026002165_0001 - Permission denied Can't create directory /disk4/yarn/nm/usercache/mcaf/appcache/application_1585026002165_0001 - Permission denied Can't create directory /disk5/yarn/nm/usercache/mcaf/appcache/application_1585026002165_0001 - Permission denied Did not create any app directories.   And i gave a trail run of my application job and that is also failing with below errors,   org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:308) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:149) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:57) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:293) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:268) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:140) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:135) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:888) ~[DMXSLoader-0.0.31.jar:0.0.31] at com.class.name.dmxsloader.main.DMXSLoaderMain.hasStagingData(DMXSLoaderMain.java:304) [DMXSLoader-0.0.31.jar:0.0.31] at com.class.name.dmxsloader.main.DMXSLoaderMain.main(DMXSLoaderMain.java:375) [DMXSLoader-0.0.31.jar:0.0.31] Caused by: java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[?:1.7.0_67] at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[?:1.7.0_67] at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[?:1.7.0_67] at sun.nio.ch.IOUtil.write(IOUtil.java:65) ~[?:1.7.0_67] at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487) ~[?:1.7.0_67] at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:63) ~[hadoop-common-2.6.0-cdh5.4.7.jar:?] at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142) ~[hadoop-common-2.6.0-cdh5.4.7.jar:?] at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:159) ~[hadoop-common-2.6.0-cdh5.4.7.jar:?] at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:117) ~[hadoop-common-2.6.0-cdh5.4.7.jar:?] at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) ~[?:1.7.0_67] at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) ~[?:1.7.0_67] at java.io.DataOutputStream.flush(DataOutputStream.java:123) ~[?:1.7.0_67] at org.apache.hadoop.hbase.ipc.IPCUtil.write(IPCUtil.java:246) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.ipc.IPCUtil.write(IPCUtil.java:234) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:895) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:850) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1184) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:216) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:300) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.get(ClientProtos.java:31865) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRowOrBefore(ProtobufUtil.java:1580) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1294) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1126) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:299) ~[DMXSLoader-0.0.31.jar:0.0.31] ... 10 more     NOTE: I have kept the below command in first line of my application script before going to launch the job, kinit -kt mcafmerged.keytab  mcaf@MWKRBCDH.ORG     Please let me know where i am missing here ?   Thanks & Regards, Vinod   ... View more

@venkatsambath Sorry for late response.   Thank you for your valuable response and i got your point where i am doing mistake. Here i want to create a keytab file for a user and that user can access all the services like, hdfs, hbase and other services running in the cluster.   I have tried with following steps please suggest me with your inputs.   sudo ktutil ktutil:  addent -password -p mcaf@Domain.ORG -k 1 -e RC4-HMAC Password for mcaf@Domain.ORG: ktutil:  wkt mcaf.keytab ktutil:  q     klist -kt mcaf.keytab Keytab name: FILE:mcaf.keytab KVNO Timestamp Principal ---- ----------------- -------------------------------------------------------- 1 03/23/20 11:58:38 mcaf@Domain.ORG   sudo kinit -kt mcaf.keytab mcaf@Domain.ORG     And able to access hdfs using, hadoop fs -ls /   But coming to hbase, i am not able to see the tables. hbase(main):001:0> list TABLE 0 row(s) in 0.4090 seconds => []     When i copied the latest keytab from process directory for hbase-master,   dayrhemwkq001:~:HADOOP QA]$ kinit -kt hbase.keytab hbase/dayrhemwkq001.enterprisenet.org@MWKRBCDH.ORG   I can able to see the tables.     My question is, I want to give a access to the user and that user can access hbase, hdfs and other services running in the luster.   Please suggest me with your inputs.   Best Regards, Vinod ... View more

Hi @venkatsambath,   As you said i have kept the kinit commands in first step in my scripts and when ever we execute the commands the kinit also run. But still i am facing same issue but this time i can see zookeeper as a user,   The commands i am using,  kinit -kt /home/mcaf/hdfs.keytab hdfs/hostname@Domain.ORG kinit -kt /home/mcaf/hdfs.keytab HTTP/hostname@Domain.ORG kinit -kt /home/mcaf/hbase.keytab hbase/hostname@Domain.ORG kinit -kt /home/mcaf/yarn.keytab HTTP/hostname@Domain.ORG kinit -kt /home/mcaf/yarn.keytab yarn/hostname@Domain.ORG kinit -kt /home/mcaf/zookeeper.keytab zookeeper/hostname@Domain.org     Error Logs,   20/03/04 02:00:42 WARN security.UserGroupInformation: PriviledgedActionException as:zookeeper/hostname@Domain.ORG (auth:KERBEROS) cause:org.apache.hadoop.security.AccessControlException: Permission denied: user=zookeeper, access=WRITE, inode="/user":mcaf:supergroup:drwxr-xr-x at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:257) at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:238) at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:216) at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:145) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:138) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6599) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6581) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:6533) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:4337) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:4307) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:4280) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:853) at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.mkdirs(AuthorizationProviderProxyClientProtocol.java:321) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:601) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038) org.apache.hadoop.security.AccessControlException: Permission denied: user=zookeeper, access=WRITE, inode="/user":mcaf:supergroup:drwxr-xr-x org.apache.hadoop.security.AccessControlException: Permission denied: user=zookeeper, access=WRITE, inode="/user":mcaf:supergroup:drwxr-xr-x at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:257) at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:238) at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:216) at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:145) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:138) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6599) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6581) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:6533) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:4337) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:4307) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:4280) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:853) at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.mkdirs(AuthorizationProviderProxyClientProtocol.java:321) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:601) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)     Can you please help me on this issue?   Best Regards, Vinod   ... View more

Thank you @venkatsambath  After modifying the min user id value to 500 i can able to run sample mapreduce job and i can see it in yarn applications in cloudera manager.   Now, I have tried with my regular job in same cluster, But it is failing and find below error messages,   ERROR 2020Feb19 02:01:21,086 main com.client.engineering.group.JOB.main.JOBMain: org.apache.hadoop.hbase.client.RetriesExhaustedException thrown: Can't get the location org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:308) ~[JOB-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:149) ~[JOB-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:57) ~[JOB-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200) ~[JOB-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:293) ~[JOB-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:268) ~[JOB-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:140) ~[JOB-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:135) ~[JOB-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:888) ~[JOB-0.0.31.jar:0.0.31] at com.client.engineering.group.JOB.main.JOBMain.hasStagingData(JOBMain.java:304) [JOB-0.0.31.jar:0.0.31] at com.client.engineering.group.JOB.main.JOBMain.main(JOBMain.java:375) [JOB-0.0.31.jar:0.0.31] Caused by: java.io.IOException: Broken pipe   ERROR 2020Feb19 02:01:30,198 main com.client.engineering.group.job.main.jobMain: _v.1.0.0a_ org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException thrown: Failed 1 action: IOException: 1 time, org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 action: IOException: 1 time,   NOTE: I have executed the kinit mcaf before executing my job.   And do we need to execute 'kinit mcaf' every time before submitting the job ? And how can we configure scheduled jobs ?   Please help me to understand.   Best Regards, Vinod ... View more

Hello @venkatsambath ,   FYI... min.user.id is set to 1000 in my Yarn configurations. allowed.system.users is set to impala,nobody,llama,hive in my Yarn configurations.   Thanks, Vinod ... View more

Hi @venkatsambath    First verified whether i am able to access hdfs before doing "kinit mcaf" and its failed to access. Now i did kinit mcaf and verified hdfs access and able to list the files and able to create a directories. Now i tried triggered sample yarn job,   hostname.com:~:HADOOP QA]$ yarn jar /opt/cloudera/parcels/CDH-5.4.7-1.cdh5.4.7.p0.3/jars/hadoop-examples.jar teragen 500000000 /tmp/teragen4 20/02/19 00:46:30 INFO client.RMProxy: Connecting to ResourceManager at resourcemanager/IP_ADDRESS:8032 20/02/19 00:46:30 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 8 for mcaf on ha-hdfs:nameservice1 20/02/19 00:46:30 INFO security.TokenCache: Got dt for hdfs://nameservice1; Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:nameservice1, Ident: (HDFS_DELEGATION_TOKEN token 8 for mcaf) 20/02/19 00:46:31 INFO terasort.TeraSort: Generating 500000000 using 2 20/02/19 00:46:31 INFO mapreduce.JobSubmitter: number of splits:2 20/02/19 00:46:31 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1582090413480_0002 20/02/19 00:46:31 INFO mapreduce.JobSubmitter: Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:nameservice1, Ident: (HDFS_DELEGATION_TOKEN token 8 for mcaf) 20/02/19 00:46:32 INFO impl.YarnClientImpl: Submitted application application_1582090413480_0002 20/02/19 00:46:32 INFO mapreduce.Job: The url to track the job: http://resourcemanager:8088/proxy/application_1582090413480_0002/ 20/02/19 00:46:32 INFO mapreduce.Job: Running job: job_1582090413480_0002 20/02/19 00:46:34 INFO mapreduce.Job: Job job_1582090413480_0002 running in uber mode : false 20/02/19 00:46:34 INFO mapreduce.Job: map 0% reduce 0% 20/02/19 00:46:34 INFO mapreduce.Job: Job job_1582090413480_0002 failed with state FAILED due to: Application application_1582090413480_0002 failed 2 times due to AM Container for appattempt_1582090413480_0002_000002 exited with exitCode: -1000 For more detailed output, check application tracking page:http://resourcemanager:8088/proxy/application_1582090413480_0002/Then, click on links to logs of each attempt. Diagnostics: Application application_1582090413480_0002 initialization failed (exitCode=255) with output: Requested user mcaf is not whitelisted and has id 779,which is below the minimum allowed 1000 Failing this attempt. Failing the application. 20/02/19 00:46:34 INFO mapreduce.Job: Counters: 0   Can you please check it and let me know please.   Regards, Vinod   ... View more

Hello @venkatsambath    Thank you for your response...!! Actually we use mcaf as a user to execute the jobs but why http user coming to the picture ?   hostname.com:~:HADOOP QA]$ groups mcaf supergroup hostname.com:~:HADOOP QA]$ users mcaf hostname.com:~:HADOOP QA]$ hadoop fs -ls / Found 4 items drwx------ - hbase supergroup 0 2020-02-18 02:46 /hbase drwxr-xr-x - hdfs supergroup 0 2015-02-04 11:44 /system drwxrwxrwt - hdfs supergroup 0 2020-02-17 05:07 /tmp drwxr-xr-x - mcaf supergroup 0 2019-03-28 03:12 /user   hostname.com:~:HADOOP QA]$ getent group supergroup supergroup:x:25290:hbase,mcaf,zookeeper,hdfs   hostname.com:~:HADOOP QA]$ getent group hadoop hadoop:x:497:mapred,yarn,hdfs     Can you please have a look and suggest me what to do? Note: I am trying to enable Kerberos and once it is running with out any interrupt or with out any issues, then we are planing to integrate with AD.   Thanks, Vinod ... View more

Hi @EricL ,   Can you please let me know your comments, And the strange thing is i have cleanedup everything and freshly installed and setup a cluster. But still we are facing same issue and the jobs are running in Local mode not in YARN Mode. I have ran it in Active Resource Manager server and edgenode, But no luck.   Can someone please let us know where i can debug and fix this issue ? Is this related to OS level of issue or YARN issue or any other issue? When i ran hostinspector i ddint find any issues like, Firewall, selinux or other issues.   Please do the needful and help us.   Best Regards, Vinod ... View more

Hi @EricL    Yes i can see same issue when i am launching the jobs in local mode.   Please find the mapred-site.xml and yarn-site.xml.   mapred-site.xml ============   -<configuration> -<property> <name>mapreduce.job.split.metainfo.maxsize</name> <value>10000000</value> </property> -<property> <name>mapreduce.job.counters.max</name> <value>120</value> </property> -<property> <name>mapreduce.job.counters.groups.max</name> <value>50</value> </property> -<property> <name>mapreduce.output.fileoutputformat.compress</name> <value>false</value> </property> -<property> <name>mapreduce.output.fileoutputformat.compress.type</name> <value>BLOCK</value> </property> -<property> <name>mapreduce.output.fileoutputformat.compress.codec</name> <value>org.apache.hadoop.io.compress.DefaultCodec</value> </property> -<property> <name>mapreduce.map.output.compress.codec</name> <value>org.apache.hadoop.io.compress.SnappyCodec</value> </property> -<property> <name>mapreduce.map.output.compress</name> <value>true</value> </property> -<property> <name>zlib.compress.level</name> <value>DEFAULT_COMPRESSION</value> </property> -<property> <name>mapreduce.task.io.sort.factor</name> <value>64</value> </property> -<property> <name>mapreduce.map.sort.spill.percent</name> <value>0.8</value> </property> -<property> <name>mapreduce.reduce.shuffle.parallelcopies</name> <value>10</value> </property> -<property> <name>mapreduce.task.timeout</name> <value>600000</value> </property> -<property> <name>mapreduce.client.submit.file.replication</name> <value>3</value> </property> -<property> <name>mapreduce.job.reduces</name> <value>1036</value> </property> -<property> <name>mapreduce.task.io.sort.mb</name> <value>256</value> </property> -<property> <name>mapreduce.map.speculative</name> <value>false</value> </property> -<property> <name>mapreduce.reduce.speculative</name> <value>false</value> </property> -<property> <name>mapreduce.job.reduce.slowstart.completedmaps</name> <value>0.8</value> </property> -<property> <name>mapreduce.jobhistory.address</name> <value>hostname.enterprisenet.org:10020</value> </property> -<property> <name>mapreduce.jobhistory.webapp.address</name> <value>hostname.enterprisenet.org:19888</value> </property> -<property> <name>mapreduce.jobhistory.webapp.https.address</name> <value>hostname.enterprisenet.org:19890</value> </property> -<property> <name>mapreduce.jobhistory.admin.address</name> <value>hostname.enterprisenet.org:10033</value> </property> -<property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> -<property> <name>yarn.app.mapreduce.am.staging-dir</name> <value>/user</value> </property> -<property> <name>mapreduce.am.max-attempts</name> <value>2</value> </property> -<property> <name>yarn.app.mapreduce.am.resource.mb</name> <value>1536</value> </property> -<property> <name>yarn.app.mapreduce.am.resource.cpu-vcores</name> <value>1</value> </property> -<property> <name>mapreduce.job.ubertask.enable</name> <value>false</value> </property> -<property> <name>yarn.app.mapreduce.am.command-opts</name> <value>-Djava.net.preferIPv4Stack=true -Xmx1073741824</value> </property> -<property> <name>mapreduce.map.java.opts</name> <value>-Djava.net.preferIPv4Stack=true -Xmx2147483648</value> </property> -<property> <name>mapreduce.reduce.java.opts</name> <value>-Djava.net.preferIPv4Stack=true -Xmx825955249</value> </property> -<property> <name>yarn.app.mapreduce.am.admin.user.env</name> <value>LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native:$JAVA_LIBRARY_PATH</value> </property> -<property> <name>mapreduce.map.memory.mb</name> <value>2560</value> </property> -<property> <name>mapreduce.map.cpu.vcores</name> <value>1</value> </property> -<property> <name>mapreduce.reduce.memory.mb</name> <value>1024</value> </property> -<property> <name>mapreduce.reduce.cpu.vcores</name> <value>1</value> </property> -<property> <name>mapreduce.application.classpath</name> <value>$HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,$MR2_CLASSPATH</value> </property> -<property> <name>mapreduce.admin.user.env</name> <value>LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native:$JAVA_LIBRARY_PATH</value> </property> </configuration>       yarn-site.xml ==========   -<configuration> -<property> <name>yarn.acl.enable</name> <value>true</value> </property> -<property> <name>yarn.admin.acl</name> <value>*</value> </property> -<property> <name>yarn.resourcemanager.ha.enabled</name> <value>true</value> </property> -<property> <name>yarn.resourcemanager.ha.automatic-failover.enabled</name> <value>true</value> </property> -<property> <name>yarn.resourcemanager.ha.automatic-failover.embedded</name> <value>true</value> </property> -<property> <name>yarn.resourcemanager.recovery.enabled</name> <value>true</value> </property> -<property> <name>yarn.resourcemanager.zk-address</name> <value>hostname_2.enterprisenet.org:2181,hostname.enterprisenet.org:2181,hostname_1.enterprisenet.org:2181</value> </property> -<property> <name>yarn.resourcemanager.store.class</name> <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value> </property> -<property> <name>yarn.client.failover-sleep-base-ms</name> <value>100</value> </property> -<property> <name>yarn.client.failover-sleep-max-ms</name> <value>2000</value> </property> -<property> <name>yarn.resourcemanager.cluster-id</name> <value>yarnRM</value> </property> -<property> <name>yarn.resourcemanager.address.rm648</name> <value>hostname_1.enterprisenet.org:8032</value> </property> -<property> <name>yarn.resourcemanager.scheduler.address.rm648</name> <value>hostname_1.enterprisenet.org:8030</value> </property> -<property> <name>yarn.resourcemanager.resource-tracker.address.rm648</name> <value>hostname_1.enterprisenet.org:8031</value> </property> -<property> <name>yarn.resourcemanager.admin.address.rm648</name> <value>hostname_1.enterprisenet.org:8033</value> </property> -<property> <name>yarn.resourcemanager.webapp.address.rm648</name> <value>hostname_1.enterprisenet.org:8088</value> </property> -<property> <name>yarn.resourcemanager.webapp.https.address.rm648</name> <value>hostname_1.enterprisenet.org:8090</value> </property> -<property> <name>yarn.resourcemanager.address.rm647</name> <value>hostname.enterprisenet.org:8032</value> </property> -<property> <name>yarn.resourcemanager.scheduler.address.rm647</name> <value>hostname.enterprisenet.org:8030</value> </property> -<property> <name>yarn.resourcemanager.resource-tracker.address.rm647</name> <value>hostname.enterprisenet.org:8031</value> </property> -<property> <name>yarn.resourcemanager.admin.address.rm647</name> <value>hostname.enterprisenet.org:8033</value> </property> -<property> <name>yarn.resourcemanager.webapp.address.rm647</name> <value>hostname.enterprisenet.org:8088</value> </property> -<property> <name>yarn.resourcemanager.webapp.https.address.rm647</name> <value>hostname.enterprisenet.org:8090</value> </property> -<property> <name>yarn.resourcemanager.ha.rm-ids</name> <value>rm648,rm647</value> </property> -<property> <name>yarn.resourcemanager.client.thread-count</name> <value>50</value> </property> -<property> <name>yarn.resourcemanager.scheduler.client.thread-count</name> <value>50</value> </property> -<property> <name>yarn.resourcemanager.admin.client.thread-count</name> <value>1</value> </property> -<property> <name>yarn.scheduler.minimum-allocation-mb</name> <value>1024</value> </property> -<property> <name>yarn.scheduler.increment-allocation-mb</name> <value>512</value> </property> -<property> <name>yarn.scheduler.maximum-allocation-mb</name> <value>102400</value> </property> -<property> <name>yarn.scheduler.minimum-allocation-vcores</name> <value>1</value> </property> -<property> <name>yarn.scheduler.increment-allocation-vcores</name> <value>1</value> </property> -<property> <name>yarn.scheduler.maximum-allocation-vcores</name> <value>56</value> </property> -<property> <name>yarn.resourcemanager.amliveliness-monitor.interval-ms</name> <value>1000</value> </property> -<property> <name>yarn.am.liveness-monitor.expiry-interval-ms</name> <value>600000</value> </property> -<property> <name>yarn.resourcemanager.am.max-attempts</name> <value>2</value> </property> -<property> <name>yarn.resourcemanager.container.liveness-monitor.interval-ms</name> <value>600000</value> </property> -<property> <name>yarn.resourcemanager.nm.liveness-monitor.interval-ms</name> <value>1000</value> </property> -<property> <name>yarn.nm.liveness-monitor.expiry-interval-ms</name> <value>600000</value> </property> -<property> <name>yarn.resourcemanager.resource-tracker.client.thread-count</name> <value>50</value> </property> -<property> <name>yarn.application.classpath</name> <value>$HADOOP_CLIENT_CONF_DIR,$HADOOP_CONF_DIR,$HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,$HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,$HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/*</value> </property> -<property> <name>yarn.resourcemanager.scheduler.class</name> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value> </property> -<property> <name>yarn.resourcemanager.max-completed-applications</name> <value>10000</value> </property> -<property> <name>yarn.nodemanager.remote-app-log-dir</name> <value>/tmp/logs</value> </property> -<property> <name>yarn.nodemanager.remote-app-log-dir-suffix</name> <value>logs</value> </property> </configuration>   Let me know if you need any other information.   Thanks, Vinod ... View more

Hello @EricL    Can you please help us what could be the reason for issue ?   Best Regards, Vinod ... View more

Hi @EricL ,   Thanks for your response,   Yes I can see Gateway roles have been assigned to the edgenode. Yes, I can able to list the files from hdfs using hdfs commands. Only YARN having the issues and we are able to access HBase Shell, Hive Shell, HDFS but Yarn we cant able to.   I have ran below sample mapreduce job,   yarn jar /opt/cloudera/parcels/CDH/ jars/hadoop-examples.jar teragen 500000000 /tmp/test5_teragen0 20/01/09 06:12:26 INFO Configuration.deprecation:  session.id  is deprecated. Instead, use dfs.metrics.session- 20/01/09 06:12:26 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId= 20/01/09 06:12:26 INFO terasort.TeraSort: Generating 500000000 using 1 20/01/09 06:12:26 INFO mapreduce.JobSubmitter: number of splits:1 20/01/09 06:12:27 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1886679574_0001 20/01/09 06:12:27 INFO mapreduce.Job: The url to track the job:  20/01/09 06:12:27 INFO mapreduce.Job: Running job:  job_local1886679574_0001   Which is not created any aplication ID.   Please respond me back and give me your suggestions.   Best Regards, Vinod ... View more

Hi @EricL    We are using Cloudera Manager,  The another clue is When ever we are submitting the jobs it was launching in local mode not yarn mode. We have a YARN gateway in Edgenode but the jobs are not launching in YARN.   When i run test run in edgenode, yarn jar /opt/cloudera/parcels/CDH/jars/hadoop-examples.jar teragen 500000000 /tmp/test5_teragen7 20/01/09 02:24:18 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id 20/01/09 02:24:18 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId= 20/01/09 02:24:19 INFO terasort.TeraSort: Generating 500000000 using 1 20/01/09 02:24:19 INFO mapreduce.JobSubmitter: number of splits:1 20/01/09 02:24:19 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1658357393_0001 20/01/09 02:24:20 INFO mapreduce.Job: The url to track the job: http://localhost:8080/ 20/01/09 02:24:20 INFO mapreduce.Job: Running job: job_local1658357393_0001 20/01/09 02:24:20 INFO mapred.LocalJobRunner: OutputCommitter set in config null 20/01/09 02:24:20 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1 20/01/09 02:24:20 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 20/01/09 02:24:20 INFO mapred.LocalJobRunner: Waiting for map tasks 20/01/09 02:24:20 INFO mapred.LocalJobRunner: Starting task: attempt_local1658357393_0001_m_000000_0 20/01/09 02:24:20 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1 20/01/09 02:24:20 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ] 20/01/09 02:24:20 INFO mapred.MapTask: Processing split: org.apache.hadoop.examples.terasort.TeraGen$RangeInputFormat$RangeInputSplit@4d182ef0 20/01/09 02:24:21 INFO mapreduce.Job: Job job_local1658357393_0001 running in uber mode : false 20/01/09 02:24:21 INFO mapreduce.Job: map 0% reduce 0%     I cant see this job in YARN Applications.   When i ran the same in other environment, i can see it is launching application and i can see it in YARN Applications.   yarn jar /opt/cloudera/parcels/CDH-5.4.10-1.cdh5.4.10.p0.16/jars/hadoop-examples.jar teragen 500000000 /tmp/test5_teragen33 20/01/09 02:27:39 INFO terasort.TeraSort: Generating 500000000 using 2 20/01/09 02:27:39 INFO mapreduce.JobSubmitter: number of splits:2 20/01/09 02:27:39 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1569582055993_48877 20/01/09 02:27:39 INFO impl.YarnClientImpl: Submitted application application_1569582055993_48877 20/01/09 02:27:39 INFO mapreduce.Job: The url to track the job: http://tpacoslmpp050.enterprisenet.org:8088/proxy/application_1569582055993_48877/ 20/01/09 02:27:39 INFO mapreduce.Job: Running job: job_1569582055993_48877 20/01/09 02:27:45 INFO mapreduce.Job: Job job_1569582055993_48877 running in uber mode : false 20/01/09 02:27:45 INFO mapreduce.Job: map 0% reduce 0% 20/01/09 02:27:55 INFO mapreduce.Job: map 2% reduce 0%   Actually i have removed YARN service and re-added and re-created Jobhistory directory and nodemanager logs directory through Cloudera Manager. We are using Cloudera Express 5.16.2 and CDH 5.4.10, Parcels.   Now also facing same issue, After re-adding the YARN to the environment.   Here we want to launch the jobs in YARN mode not local mode. Please do the needful.   Thanks, Vinod   ... View more

Hello Team,   Any updates would be appreciated and still we are seeing same problem.   Thanks, Vinod  ... View more

Hello @EricL    Can you please help me, There is no job running in YARN and getting below warnings and errors in NODEMANAGER,     2020-01-07 11:07:45,764 ERROR org.apache.hadoop.yarn.server.nodemanager.NodeManager: RECEIVED SIGNAL 15: SIGTERM 2020-01-07 11:07:45,977 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl is interrupted. Exiting. 2020-01-07 11:07:59,163 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: The Auxilurary Service named 'mapreduce_shuffle' in the configuration is for class class org.apache.hadoop.mapred.ShuffleHandler which has a name of 'httpshuffle'. Because these are not the same tools trying to send ServiceData and read Service Meta Data may have issues unless the refer to the name in the config.         Resource Manager Logs, ===================     2020-01-07 10:18:59,364 ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: RECEIVED SIGNAL 15: SIGTERM 2020-01-07 10:18:59,371 ERROR org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted 2020-01-07 10:18:59,375 ERROR org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted 2020-01-07 10:18:59,377 ERROR org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted 2020-01-07 10:18:59,494 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x16f807d0eea02cd 2020-01-07 10:18:59,499 WARN org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher: org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread interrupted. Returning. 2020-01-07 10:18:59,517 ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Returning, interrupted : java.lang.InterruptedException 2020-01-07 10:18:59,518 WARN org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Update thread interrupted. Exiting. 2020-01-07 10:18:59,519 ERROR org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted     Job History Server : ===============   2020-01-07 11:08:14,980 WARN org.apache.hadoop.yarn.logaggregation.AggregatedLogDeletionService: Could not read the contents of hdfs://nameservice1/tmp/logs/mcaf/logs Permission denied: user=mapred, access=READ_EXECUTE, inode="/tmp/logs/mcaf/logs":mcaf:supergroup:drwxrwx--- 2020-01-07 11:08:14,980 INFO org.apache.hadoop.yarn.logaggregation.AggregatedLogDeletionService: aggregated log deletion finished. 2020-01-07 11:08:15,038 INFO org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://nameservice1:8020] 2020-01-07 11:08:43,242 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: History Cleaner started 2020-01-07 11:08:43,348 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: History Cleaner complete 2020-01-07 11:11:13,239 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files     Thanks in advance.....!!!   Best Regards, Vinod       ... View more