Solved this issue.   My problem was simple, My NodeManager log directory setting is not pointing to the container.   yarn.nodemanager.log-dirs=/apps/ext/var/log/hadoop-yarn   it was suppopsed to be   yarn.nodemanager.log-dirs=/apps/ext/var/log/hadoop-yarn/container   I figured it out after I see this message on my NodeManager web ui, under Node Log information, "NodeHealthReport 1/1 log-dirs turned bad"   I was able to run and finish the PiEstimator after I update the log-dirs and restart the service. ... View more

Please disregard this post. Dumb post. I didn't realize those arrows for each category can be expanded, which gives me the ability to change the log directory.   Sorry. ... View more