I have a very similar issue. After integratin the entire cluster, all services including Ranger, with the Active Directory, the YARN application log folders for Active Directory users are empty.
This does not happen when the application is run as the hdfs user, so it has to be a permissions problem.
While an application is running I can see the log folders and files on the node running the container for the application:
/opt/hadoop/yarn/log/application_1548066747260_2576
/opt/hadoop/yarn/log/application_1548066747260_2576/container_e71_1548066747260_2576_01_000001
/opt/hadoop/yarn/log/application_1548066747260_2576/container_e71_1548066747260_2576_01_000001/launch_container.sh
/opt/hadoop/yarn/log/application_1548066747260_2576/container_e71_1548066747260_2576_01_000001/directory.info
As soon as the application stops the log folder is created in hdfs, but it is empty.
The Ranger policies have been set up to give service accounts (e.g. hdfs, hive, spark etc.) full access to the entire hdfs file structure.
AD users who are allowed to submit spark jobs are given access to these folders in an attempt to allow application logs to work:
/tmp
/mr-history
/ats
/spark-history
/spark2-history
/livy-recovery
/livy2-recovery
/app-logs
/app-logs/oozie
/app-logs/{USER}
/user/{USER}
Any suggestions will be greatly appreciated.
I can post more configuration and settings information if needed.
... View more
