Support Questions

Find answers, ask questions, and share your expertise

YARN job is Running Slow

Explorer

Hello Everyone

i am running one YARN job on daily basis ,but today its running little bit slow than yesterday, on the same node other peoples also running the same job but they didn't face any issue regarding the slowness job,

only my job is running slow and its a daily running job

what could be the possible reason behind this?

 

thanks in advance !!

1 ACCEPTED SOLUTION

Rising Star

Hi @Nil_kharat 

 

If you are running a hive job and if you noticed it is slower than the earlier run then you may need to check few things in the below order.

 

1. Is there any change in the code,
2. Is there any new data load added to the table which is getting processed,
3. Beeline console output where you can see the Hive counters details,
4. Hiveserver2 and Hivemetastore logs to see the compilation and execution time of the faster and slower run.
5. QueryId of the job(to check in HS2 and application logs)
6. Faster and slower run of application logs where identify the slowness.
7. explain <query> for the faster and slower(to see the record change and stats info)


If you are happy with the response mark it as Accept as Solution

View solution in original post

8 REPLIES 8

Rising Star

Hi @Nil_kharat 

 

If you are running a hive job and if you noticed it is slower than the earlier run then you may need to check few things in the below order.

 

1. Is there any change in the code,
2. Is there any new data load added to the table which is getting processed,
3. Beeline console output where you can see the Hive counters details,
4. Hiveserver2 and Hivemetastore logs to see the compilation and execution time of the faster and slower run.
5. QueryId of the job(to check in HS2 and application logs)
6. Faster and slower run of application logs where identify the slowness.
7. explain <query> for the faster and slower(to see the record change and stats info)


If you are happy with the response mark it as Accept as Solution

Explorer

hi @Shifu I have checked everything , didnt get any solution, some jobs are in ACCEPTED state, how to troubleshoot this accepted state jobs, can we move particular queue, or any other? 

 

 

 

 

Thanks

Rising Star

Hi @Nil_kharat 

 

If your jobs are at ACCEPTED most probably it is because the AM does not have enough memory to launch in that particular queue.

 

You can click on top of the particular ACCEPTED job to see the detail.
Can you try to increase the Maximum AM Resource(Ambari > Tile Icon > Yarn Queue Manager > Particular queue) to 50% and try to re-run the query and check.

If you are happy with the response mark it as Accept as Solution

Community Manager

@Nil_kharat Have you resolved your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. 

cjervis_0-1629405986667.png

 

 


Cy Jervis, Manager, Community Program
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Explorer

Hi @Shifu im using cloudera manager cdh5.16

Still not resolved the issue.

What are the some ways that we can solve it 

From cloudera manager 

 

And one more thing how can we track the job that are running by user's.

How can we track the status from cli or gui

 

 

 

 

 

 

 

 

 

 

 

 

Explorer

Go to your job from YARN UI >> and check the logs , it's good also to check the container logs .
also make sure the there is queue mapping for your user ; and you are having resources for your job .

 

From the cli

yarn logs -applicationId <your application ID>

 

Rising Star

Hi @Nil_kharat 

Still not resolved the issue.
You may need to check the HS2 logs and application logs to figure the slowness.


And one more thing how can we track the job that are running by user's.
1. Go to RM UI > Running/finished/killed > check the User column
2. CM > YARN > Applications > Based upon the user you can search over here.

If you are happy with the response mark it as Accepts as Solution

 

Explorer

@Shifu  if this is the case i think that he will notice a msg shows an exception if related .

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.