Hello,   Resource manager is giving warning and exiting " Name dr.who is converted to dr_dot_who when it is used as a queue name."   any possible solution ? complete log for warnings is below   3:08:49.689 PM WARN QueuePlacementRule Name dr.who is converted to dr_dot_who when it is used as a queue name. 3:08:49.689 PM INFO FairScheduler Accepted application application_1533729822715_0025 from user: dr.who, in queue: root.users.dr_dot_who, currently num of applications: 21 3:08:49.690 PM INFO RMAppImpl application_1533729822715_0025 State change from SUBMITTED to ACCEPTED on event = APP_ACCEPTED 3:08:49.690 PM INFO ApplicationMasterService Registering app attempt : appattempt_1533729822715_0025_000001 3:08:49.690 PM INFO RMAppAttemptImpl appattempt_1533729822715_0025_000001 State change from NEW to SUBMITTED on event = START 3:08:49.690 PM INFO FairScheduler Added Application Attempt appattempt_1533729822715_0025_000001 to scheduler from user: dr.who 3:08:49.691 PM INFO RMAppAttemptImpl appattempt_1533729822715_0025_000001 State change from SUBMITTED to SCHEDULED on event = ATTEMPT_ADDED 3:08:50.820 PM WARN RMAppImpl The specific max attempts: 3 for application: 7 is invalid, because it is out of the range [1, 2]. Use the global max attempts instead. 3:08:50.820 PM INFO ClientRMService Application with id 7 submitted by user dr.who 3:08:50.820 PM INFO RMAuditLogger USER=dr.who OPERATION=Submit Application Request TARGET=ClientRMService RESULT=SUCCESS APPID=application_1533729822715_0007 3:08:50.821 PM INFO RMAppImpl Storing application with id application_1533729822715_0007 3:08:50.822 PM INFO RMAppImpl application_1533729822715_0007 State change from NEW to NEW_SAVING on event = START 3:08:50.822 PM INFO RMStateStore Storing info for app: application_1533729822715_0007 3:08:50.822 PM INFO RMAppImpl application_1533729822715_0007 State change from NEW_SAVING to SUBMITTED on event = APP_NEW_SAVED 3:08:50.822 PM WARN QueuePlacementRule Name dr.who is converted to dr_dot_who when it is used as a queue name. ... View more

Hello,   Ok, so i had Node manager running completly fine and suprisingly it started to crash and exited every few minutes. For instance its exited at x time and minutes, after 10-15 minutes it will be back again.   I looked up to host logs and Node manager logs specifically, i found following message related to "stop instruction by container for application xxxx"   2018-08-06 23:10:09,842 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 3048 for container-id container_1533576341741_0986_01_000001: -1B of 1 GB physical memory used; -1B of 2.1 GB virtual memory used 2018-08-06 23:10:10,178 ERROR org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.RecoveredContainerLaunch: Unable to recover container container_1533576341741_0986_01_000001 java.io.IOException: Timeout while waiting for exit code from container_1533576341741_0986_01_000001 at org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor.reacquireContainer(ContainerExecutor.java:199) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.RecoveredContainerLaunch.call(RecoveredContainerLaunch.java:83) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.RecoveredContainerLaunch.call(RecoveredContainerLaunch.java:46) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 2018-08-06 23:10:10,186 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.RecoveredContainerLaunch: Recovered container exited with a non-zero exit code 154 2018-08-06 23:10:10,191 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1533576341741_0986_01_000001 transitioned from RUNNING to EXITED_WITH_FAILURE 2018-08-06 23:10:10,191 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_1533576341741_0986_01_000001 2018-08-06 23:10:10,259 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /yarn/nm/usercache/dr.who/appcache/application_1533576341741_0986/container_1533576341741_0986_01_000001 2018-08-06 23:10:10,270 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=dr.who OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: EXITED_WITH_FAILURE APPID=application_1533576341741_0986 CONTAINERID=container_1533576341741_0986_01_000001 2018-08-06 23:10:10,278 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1533576341741_0986_01_000001 transitioned from EXITED_WITH_FAILURE to DONE 2018-08-06 23:10:10,279 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Removing container_1533576341741_0986_01_000001 from application application_1533576341741_0986 2018-08-06 23:10:10,280 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_1533576341741_0986_01_000001 for log-aggregation 2018-08-06 23:10:10,280 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1533576341741_0986 2018-08-06 23:10:11,287 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_1533576341741_0986_01_000001] 2018-08-06 23:10:12,843 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_1533576341741_0986_01_000001     Any one faced similar issue ? or can help me solve it ?   Thanks ... View more