How frequently is this occurring? It is worth checking network drops or you might consider increasing the timeouts on both sides ie. Resource Manager and Zookeeper. ... View more

Can you isolate any connection issues between your NN and DN pods? Maybe you can try doing an nc or telnet to the NN port from the DN pod? ... View more

Did you get to check the application logs? This should give you more information to determine the exact flow. ... View more

In case you have a valid kerberos ticket, kindly use --negotiate and submit again. H!nt curl -X GET "https://master-servere.com:8090/ws/v43/cluster/apps/new-application" -H "accept: application/json" -k -v --negotiate -u admin   You may check for a valid Kerberos ticket using #klist  ... View more

Please share a screenshot of your placement rules ... View more

Have you checked for any application initiation being done within the code that you are trying to submit? ... View more

Try analyzing the jstacks for the container in a stuck state. To identify and capture the jstacks for a stuck container you may follow this article  ... View more

It is worth looking at the application logs here and checking user permissions for your queue here ... View more

In addition to looking at the application logs here You can check for user permissions for your queue a. In the "Queue Details" section (YARN Queue Manager UI), scroll down to the "Access Control" subsection. In the "Access Control" section, you will see a table that lists the users and groups that have been granted permissions to submit jobs to the queue, and the type of permissions that have been granted (e.g. "Submit Application", "Administer Queue", etc.). ... View more

I see a disabled log aggregation for these applications. You can enable the log aggregation and look for possible reasons in the application logs. To view the entire log for an application yarn logs -applicationId <application-id> Below is a sample snippet of yarn-site.xml to enable log aggregation. You can look up or add these properties in the Hadoop management application (like cloudera-manager) as well. <property> <name>yarn.log-aggregation-enable</name> <value>true</value> </property> <property> <name>yarn.log-aggregation.retain-seconds</name> <value>604800</value> </property> <property> <name>yarn.log-aggregation.retain-check-interval-seconds</name> <value>3600</value> </property> ... View more