- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
YARN Applications display wrong formatted duration
- Labels:
-
Apache YARN
Created on ‎01-15-2018 11:07 AM - edited ‎09-16-2022 05:44 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello,
I am having a problem that I can't find any logical solution. Every job that requires YARN it will show up in "YARN Applications" UI on Cloudera Manager. Even though I can see all the running jobs on YARN Applications UI, ResourceManager UI, or Spark UI I have to widen my time selector to a year or two to see the finished jobs.
I think this has something to do with displayed time. All the running jobs have the static `17540.7d` as their duration:
At the same time these applications on `ResourceManager` are showing up with the right date/time:
As you can see this makes it really hard to monitor and track anything in YARN Applications view in Cloudera Manager.
Cloudera Manager express: 5.13.1
CDH: 5.13.1
Ubuntu Server 16.04
And I checked all the machines date/time to see if they are not sync. But unfortunately I can't find any issue in my cluster.
NOTE: there is only one similar issue here, but I guess he can't see any jobs even by
widening time window. (I can see jobs with wider time window 1-2yrs)
Best,
Maziyar
Created on ‎01-11-2019 10:51 AM - edited ‎01-11-2019 10:55 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Maziyar and Li,
You are definitely on the right track here. I see that the top-level root queue has "aclAdministerApps=maziyar,admin", which limits YARN API access to some of the time metrics for the application. If you're not one of these users when you make the API call you won't get correct value returned for:
- startedTime
- finishedTime
- elapsedTime
- logAggregationStatus
- amHostHttpAddress
- usedResources
- allocatedMB
- allocatedVCores
- runningContainers
Although you will get some basic application info, so you'll see the application, but metrics will be wrong, just like you report.
So, the question is which user is the CM service using to interact with the YARN API. I did some testing and reproduced your issue by limiting the aclAdministerApps property to "yarn". I then found that when I add "dr.who" to aclAdministerApps at the root level, it starts working properly.
So, try modifying your root level ACLs to be "aclAdministerApps=maziyar,admin,dr.who", refresh the Dynamic Resource Pool (DRP) configuration, and see if it resolves the issue for you.
Nick
Created ‎01-16-2019 07:41 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks for the advice. I have added "dr.who" to the list and now everything is back to normal! Many thanks mate 🙂
Created ‎01-17-2019 07:52 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Maziyar,
I was discussing this issue internally and adding "dr.who" to the adminACL has the side effect of allowing all users to have access, so we don't want that. I know we're on the right track here, we just need to get the correct user or group added to the adminACL for CM. I'm researching and will update as soon as I have the answer!
Nick
Created ‎01-17-2019 08:17 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Created ‎01-18-2019 08:27 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Maziyar,
I've found that CM uses the "hue" user to interact with the YARN API, so try changing the root level ACL to be "aclAdministerApps=maziyar,admin,hue", refresh the Dynamic Resource Pool (DRP) configuration, and test if it still resolves the issue for you. This will be much more restricted than using "dr.who" but allow the CM Web UI to function properly.
Nick
Created ‎01-18-2019 09:22 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Unfortunately, removing dr.who and adding hue resulted in the same problem as I had initially. I do agree to add hue would be much safer and restricted than dr.who, but it didn't work.
I am looking forward to something similar to hue to solve this issue 🙂
Many thanks,
Maziyar
Created ‎01-21-2019 11:15 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Maziyar,
I'm digging in to this again. I clearly see messages in the RM log showing "dr.who" is the user accessing the YARN API. I'm researching further so I hopefully can provide the correct answer!
Nick
Created on ‎01-25-2019 08:45 AM - edited ‎01-25-2019 09:00 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Maziyar,
The information I had about user "hue" being used by CM to access YARN API is correct for kerberized clusters, but in your case we know that the cluster is not kerberized and we see "dr.who" is used by CM. Consequently, I think that adding "dr.who" to the aclAdminsterApps property is the only solution for now.
I am creating an internal improvement request for Cloudera Manager (CM) to also use the use "hue" if ACLs are turned on in a non-kerberized cluster. That way the behavior will be consistent and will provide some level of restriction on who can administer queues in a non-kerberized environment.
EDIT: Internal Improvement JIRA created for CM - look for this change in a future release of CDH (no guarantees, but I hope we implement this change)
Nick
Created ‎02-01-2019 01:13 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
That would be great! Thank you so much for your time and helping me in this matter, I really appreciate it 🙂

- « Previous
-
- 1
- 2
- Next »