I have yarn.log-aggregation-enable set to true and yarn.nodemanager.remote-app-log-dir set to /app-logs
Once the job finished I don't see the directory /app-logs in hdfs. I am unable to view the job logs through resource manager. The yarn.log-aggregation.retain-seconds is set to high number. Any ideas?
@Sunile Manjee Check the NM and RM logs and grep for "aggrega". There are often times errors related to aggregation. You may need to create the /app-logs dir in hdfs and ensure proper ownership. I would start by making sure the aggregation thread is running and seen in the log. If it is, then it is a filesystem level issue.