Hi @willx , Is there a way to see if the hadoop path is a volume or a directory? ... View more

@Faizan_Ali Thanks for the explanation. Makes sense. So while an application is running, it logs the container logs into a local directory "$ {yarn.nodemanager.log-dirs}/application_${appid}" then after the application is completed, it aggregates the logs into yarn.nodemanager.remote-app-log-dir.   Ok thanks for the explanation. ... View more

Thanks @Faizan_Ali for the explanation. So in other words, once the job completes, then these logs will be stored in HDFS, or where are the logs stored after the application job is completed?   Are the local and log yarn dirs are only for temporary use, usually when a job runs in the hadoop cluster? Thanks, ... View more

Yes, I have copied tables from another cluster to this current cluster. ... View more

Also here is the application logs. [hive@sunnymaster01 ~]$ cat application_1629805664278_0004.log End of LogType:prelaunch.err ****************************************************************************** Container: container_1629805664278_0004_02_000001 on sunnydn01.dmicorp.com_45454 LogAggregationType: AGGREGATED ================================================================================ LogType:prelaunch.out LogLastModifiedTime:Wed Sep 08 22:52:35 -0500 2021 LogLength:70 LogContents: Setting up env variables Setting up job resources Launching container End of LogType:prelaunch.out ****************************************************************************** Container: container_1629805664278_0004_02_000001 on sunnydn01.dmicorp.com_45454 LogAggregationType: AGGREGATED ================================================================================ LogType:stderr LogLastModifiedTime:Wed Sep 08 22:52:35 -0500 2021 LogLength:77 LogContents: Error: Could not find or load main class org.apache.tez.dag.app.DAGAppMaster End of LogType:stderr *********************************************************************** Container: container_1629805664278_0004_02_000001 on sunnydn01.dmicorp.com_45454 LogAggregationType: AGGREGATED ================================================================================ LogType:stdout LogLastModifiedTime:Wed Sep 08 22:52:35 -0500 2021 LogLength:723 LogContents: Heap PSYoungGen total 149504K, used 5140K [0x00000000e6700000, 0x00000000f0d80000, 0x0000000100000000) eden space 128512K, 4% used [0x00000000e6700000,0x00000000e6c05208,0x00000000ee480000) from space 20992K, 0% used [0x00000000ef900000,0x00000000ef900000,0x00000000f0d80000) to space 20992K, 0% used [0x00000000ee480000,0x00000000ee480000,0x00000000ef900000) ParOldGen total 341504K, used 0K [0x00000000b3400000, 0x00000000c8180000, 0x00000000e6700000) object space 341504K, 0% used [0x00000000b3400000,0x00000000b3400000,0x00000000c8180000) Metaspace used 2971K, capacity 4550K, committed 4864K, reserved 1056768K class space used 316K, capacity 386K, committed 512K, reserved 1048576K End of LogType:stdout *********************************************************************** End of LogType:prelaunch.err ****************************************************************************** Container: container_1629805664278_0004_01_000001 on sunnydn05.dmicorp.com_45454 LogAggregationType: AGGREGATED ================================================================================ LogType:prelaunch.out LogLastModifiedTime:Wed Sep 08 22:52:35 -0500 2021 LogLength:70 LogContents: Setting up env variables Setting up job resources Launching container End of LogType:prelaunch.out ****************************************************************************** Container: container_1629805664278_0004_01_000001 on sunnydn05.dmicorp.com_45454 LogAggregationType: AGGREGATED ================================================================================ LogType:stderr LogLastModifiedTime:Wed Sep 08 22:52:35 -0500 2021 LogLength:77 LogContents: Error: Could not find or load main class org.apache.tez.dag.app.DAGAppMaster End of LogType:stderr *********************************************************************** Container: container_1629805664278_0004_01_000001 on sunnydn05.dmicorp.com_45454 LogAggregationType: AGGREGATED ================================================================================ LogType:stdout LogLastModifiedTime:Wed Sep 08 22:52:35 -0500 2021 LogLength:723 LogContents: Heap PSYoungGen total 149504K, used 5140K [0x00000000e6700000, 0x00000000f0d80000, 0x0000000100000000) eden space 128512K, 4% used [0x00000000e6700000,0x00000000e6c05208,0x00000000ee480000) from space 20992K, 0% used [0x00000000ef900000,0x00000000ef900000,0x00000000f0d80000) to space 20992K, 0% used [0x00000000ee480000,0x00000000ee480000,0x00000000ef900000) ParOldGen total 341504K, used 0K [0x00000000b3400000, 0x00000000c8180000, 0x00000000e6700000) object space 341504K, 0% used [0x00000000b3400000,0x00000000b3400000,0x00000000c8180000) Metaspace used 2973K, capacity 4550K, committed 4864K, reserved 1056768K class space used 316K, capacity 386K, committed 512K, reserved 1048576K End of LogType:stdout ***********************************************************************   Any help is much appreciated.   Thanks, ... View more