YARN Applications display wrong formatted duration

[#1]

Hello,

I am having a problem that I can't find any logical solution. Every job that requires YARN it will show up in "YARN Applications" UI on Cloudera Manager. Even though I can see all the running jobs on YARN Applications UI, ResourceManager UI, or Spark UI I have to widen my time selector to a year or two to see the finished jobs.

I think this has something to do with displayed time. All the running jobs have the static `17540.7d` as their duration:

At the same time these applications on `ResourceManager` are showing up with the right date/time:

As you can see this makes it really hard to monitor and track anything in YARN Applications view in Cloudera Manager.

Cloudera Manager express: 5.13.1

CDH: 5.13.1

Ubuntu Server 16.04

And I checked all the machines date/time to see if they are not sync. But unfortunately I can't find any issue in my cluster.

NOTE: there is only one similar issue here, but I guess he can't see any jobs even by

widening time window. (I can see jobs with wider time window 1-2yrs)

http://community.cloudera.com/t5/Batch-Processing-and-Workflow/Completed-YARN-applications-not-visib...

Best,

Maziyar

[#2]

Hi Nick,

Thanks for the advice. I have added "dr.who" to the list and now everything is back to normal! Many thanks mate 🙂

[#3]

Maziyar,

I was discussing this issue internally and adding "dr.who" to the adminACL has the side effect of allowing all users to have access, so we don't want that.  I know we're on the right track here, we just need to get the correct user or group added to the adminACL for CM.  I'm researching and will update as soon as I have the answer!

Nick

[#4]

Fantastic! Many thanks Nick and looking forward to the right solution 🙂

[#5]

Hi Maziyar,

I've found that CM uses the "hue" user to interact with the YARN API, so try changing the root level ACL to be "aclAdministerApps=maziyar,admin,hue", refresh the Dynamic Resource Pool (DRP) configuration, and test if it still resolves the issue for you.  This will be much more restricted than using "dr.who" but allow the CM Web UI to function properly.

Nick

[#6]

Hi Nick,

Unfortunately, removing dr.who and adding hue resulted in the same problem as I had initially. I do agree to add hue would be much safer and restricted than dr.who, but it didn't work.

I am looking forward to something similar to hue to solve this issue 🙂

Many thanks,
Maziyar

[#7]

Hi Maziyar,

I'm digging in to this again.  I clearly see messages in the RM log showing "dr.who" is the user accessing the YARN API.  I'm researching further so I hopefully can provide the correct answer!

Nick

[#8]

Hi Maziyar,

The information I had about user "hue" being used by CM to access YARN API is correct for kerberized clusters, but in your case we know that the cluster is not kerberized and we see "dr.who" is used by CM.  Consequently, I think that adding "dr.who" to the aclAdminsterApps property is the only solution for now.

I am creating an internal improvement request for Cloudera Manager (CM) to also use the use "hue" if ACLs are turned on in a non-kerberized cluster.  That way the behavior will be consistent and will provide some level of restriction on who can administer queues in a non-kerberized environment.

EDIT: Internal Improvement JIRA created for CM - look for this change in a future release of CDH (no guarantees, but I hope we implement this change)

Nick

[#9]

Hi Nick,

That would be great! Thank you so much for your time and helping me in this matter, I really appreciate it 🙂