Support Questions

Find answers, ask questions, and share your expertise

oozie coordinator stuck in running state

avatar
Rising Star

5339-capture-du-2016-06-29-111355.png

5338-capture-du-2016-06-29-111228.png

i create coordinator and i scheduled it to run every 20 minutes . it work fine for the beginning .after a while , i try to test it another time but it stuck in running state .

oozie message
ACCEPTED: waiting for AM container to be allocated, launched and register with RM
log coordinator
WARN CoordActionReadyXCommand:523 - SERVER[node4.bigdatau] USER[root] GROUP[-] TOKEN[] APP[every-fifteen-minutes] JOB[0000000-160629100441216-oozie-oozi-C] ACTION[] No actions to start for jobId=0000000-160629100441216-oozie-oozi-C as max concurrency reached!
log workflow
WARN ResumeXCommand:523 - SERVER[node4.bigdatau] USER[root] GROUP[-] TOKEN[] APP[WorkFlowForPigAction] JOB[0000001-160629100441216-oozie-oozi-W] ACTION[] E1100: Command precondition does not hold before execution, [workflow's status is RUNNING is not SUSPENDED], Error Code: E1100

5336-capture-du-2016-06-29-110547.png

5337-capture-du-2016-06-29-110533.png

2 REPLIES 2

avatar
Super Guru

@nejm hadjmbarek

Seems your job is submitted to Yarn and is accepted too but waiting for AM container to allocate. Seems resource constains issue here.

Can you check free memory on Yarn UI Also please check below links which might help -

http://stackoverflow.com/questions/34467308/mapreduce-job-hangs-waiting-for-am-container-to-be-alloc...

https://community.hortonworks.com/questions/27334/mapreduce-job-hang-waiting-for-am-container-to-be....

avatar
Expert Contributor

@nejm hadjmbarek, in the information you provided, it seems your oozie max concurrency has been reached for the coordinator. You therefore have a number of applications waiting AM resources. Check you max AM resource percentage in capacity scheduler and consider raising it to either .5 or .6 which states that of the total resources, RM can assign our 50 or 60 percent to AM containers.