YARN job is Running Slow

[#1]

Hello Everyone

i am running one YARN job on daily basis ,but today its running little bit slow than yesterday, on the same node other peoples also running the same job but they didn't face any issue regarding the slowness job,

only my job is running slow and its a daily running job

what could be the possible reason behind this?

thanks in advance !!

[#2 · Accepted solution]

Hi @Nil_kharat 

If you are running a hive job and if you noticed it is slower than the earlier run then you may need to check few things in the below order.

1. Is there any change in the code,
2. Is there any new data load added to the table which is getting processed,
3. Beeline console output where you can see the Hive counters details,
4. Hiveserver2 and Hivemetastore logs to see the compilation and execution time of the faster and slower run.
5. QueryId of the job(to check in HS2 and application logs)
6. Faster and slower run of application logs where identify the slowness.
7. explain <query> for the faster and slower(to see the record change and stats info)

If you are happy with the response mark it as Accept as Solution

[#3]

hi @Shifu I have checked everything , didnt get any solution, some jobs are in ACCEPTED state, how to troubleshoot this accepted state jobs, can we move particular queue, or any other? 

Thanks

[#4]

Hi @Nil_kharat 

If your jobs are at ACCEPTED most probably it is because the AM does not have enough memory to launch in that particular queue.

You can click on top of the particular ACCEPTED job to see the detail.
Can you try to increase the Maximum AM Resource(Ambari > Tile Icon > Yarn Queue Manager > Particular queue) to 50% and try to re-run the query and check.

If you are happy with the response mark it as Accept as Solution

[#5]

@Nil_kharat Have you resolved your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. 

[#6]

Hi @Shifu im using cloudera manager cdh5.16

Still not resolved the issue.

What are the some ways that we can solve it 

From cloudera manager 

And one more thing how can we track the job that are running by user's.

How can we track the status from cli or gui

[#7]

Go to your job from YARN UI >> and check the logs , it's good also to check the container logs .
also make sure the there is queue mapping for your user ; and you are having resources for your job .

From the cli

yarn logs -applicationId <your application ID>

[#8]

Hi @Nil_kharat 

Still not resolved the issue.
You may need to check the HS2 logs and application logs to figure the slowness.

And one more thing how can we track the job that are running by user's.
1. Go to RM UI > Running/finished/killed > check the User column
2. CM > YARN > Applications > Based upon the user you can search over here.

If you are happy with the response mark it as Accepts as Solution

[#9]

@Shifu  if this is the case i think that he will notice a msg shows an exception if related .