question Re: YARN job is Running Slow in Support Questions

YARN job is Running Slow

Nil_kharat — Fri, 13 Aug 2021 15:30:09 GMT

Hello Everyone

i am running one YARN job on daily basis ,but today its running little bit slow than yesterday, on the same node other peoples also running the same job but they didn't face any issue regarding the slowness job,

only my job is running slow and its a daily running job

what could be the possible reason behind this?

thanks in advance !!

Re: YARN job is Running Slow

Shifu — Mon, 16 Aug 2021 05:43:43 GMT

Hi @Nil_kharat

If you are running a hive job and if you noticed it is slower than the earlier run then you may need to check few things in the below order.

1. Is there any change in the code,
2. Is there any new data load added to the table which is getting processed,
3. Beeline console output where you can see the Hive counters details,
4. Hiveserver2 and Hivemetastore logs to see the compilation and execution time of the faster and slower run.
5. QueryId of the job(to check in HS2 and application logs)
6. Faster and slower run of application logs where identify the slowness.
7. explain <query> for the faster and slower(to see the record change and stats info)

If you are happy with the response mark it as Accept as Solution

Re: YARN job is Running Slow

Nil_kharat — Mon, 16 Aug 2021 11:33:37 GMT

hi @Shifu I have checked everything , didnt get any solution, some jobs are in ACCEPTED state, how to troubleshoot this accepted state jobs, can we move particular queue, or any other?

Thanks

Re: YARN job is Running Slow

Shifu — Mon, 16 Aug 2021 16:06:47 GMT

Hi @Nil_kharat

If your jobs are at ACCEPTED most probably it is because the AM does not have enough memory to launch in that particular queue.

You can click on top of the particular ACCEPTED job to see the detail.
Can you try to increase the Maximum AM Resource(Ambari > Tile Icon > Yarn Queue Manager > Particular queue) to 50% and try to re-run the query and check.

If you are happy with the response mark it as Accept as Solution

Re: YARN job is Running Slow

cjervis — Thu, 19 Aug 2021 20:46:39 GMT

@Nil_kharat Have you resolved your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.

Re: YARN job is Running Slow

Nil_kharat — Sat, 21 Aug 2021 04:11:34 GMT

Hi @Shifu im using cloudera manager cdh5.16

Still not resolved the issue.

What are the some ways that we can solve it

From cloudera manager

And one more thing how can we track the job that are running by user's.

How can we track the status from cli or gui

Re: YARN job is Running Slow

Moh-AJ — Sat, 21 Aug 2021 23:04:46 GMT

@Shifu if this is the case i think that he will notice a msg shows an exception if related .

Re: YARN job is Running Slow

Moh-AJ — Sat, 21 Aug 2021 23:10:21 GMT

Go to your job from YARN UI >> and check the logs , it's good also to check the container logs .
also make sure the there is queue mapping for your user ; and you are having resources for your job .

From the cli

yarn logs -applicationId <your application ID>

Re: YARN job is Running Slow

Shifu — Sun, 22 Aug 2021 07:09:01 GMT

Hi @Nil_kharat

Still not resolved the issue.
You may need to check the HS2 logs and application logs to figure the slowness.

And one more thing how can we track the job that are running by user's.
1. Go to RM UI > Running/finished/killed > check the User column
2. CM > YARN > Applications > Based upon the user you can search over here.

If you are happy with the response mark it as Accepts as Solution