Reply
Highlighted
Contributor
Posts: 26
Registered: ‎11-04-2016

YARN Applications display wrong formatted duration

[ Edited ]

Hello,

 

I am having a problem that I can't find any logical solution. Every job that requires YARN it will show up in "YARN Applications" UI on Cloudera Manager. Even though I can see all the running jobs on YARN Applications UI, ResourceManager UI, or Spark UI I have to widen my time selector to a year or two to see the finished jobs.

 

I think this has something to do with displayed time. All the running jobs have the static `17540.7d` as their duration:

 

Screenshot 2018-01-09 18.14.26.png

 

At the same time these applications on `ResourceManager` are showing up with the right date/time:

 

Screenshot 2018-01-15 19.58.56.png

As you can see this makes it really hard to monitor and track anything in YARN Applications view in Cloudera Manager.

 

Cloudera Manager express: 5.13.1

CDH: 5.13.1

Ubuntu Server 16.04

And I checked all the machines date/time to see if they are not sync. But unfortunately I can't find any issue in my cluster.

 

 

NOTE: there is only one similar issue here, but I guess he can't see any jobs even by

widening time window. (I can see jobs with wider time window 1-2yrs)

http://community.cloudera.com/t5/Batch-Processing-and-Workflow/Completed-YARN-applications-not-visib...

 

Best,

Maziyar

 
Contributor
Posts: 26
Registered: ‎11-04-2016

Re: YARN Applications display wrong formatted duration

I clear all the logs and previous jobs but the CM still have all the finished jobs with the wrong date. Also, still shows the new apps with that weird duration (which it looks like the converting the milliseconds to another time format went wrong).

 

Does anybody know where is this data coming from? I have a MySQL setup for my CM. Can I look for this to see if this is a front-end issue or back-end or being inserted into file/table wrongly from the beginning.

 

Many thanks.

Contributor
Posts: 26
Registered: ‎11-04-2016

Re: YARN Applications display wrong formatted duration

[ Edited ]

If I export the current job it shows me this:

 

  "applications" : [ {
    "applicationId" : "application_1516618738289_0001",
    "name" : "livy-session-0",
    "startTime" : "1970-01-01T00:00:00.000Z",
    "user" : "maziyar",
    "pool" : "root.users.maziyar",
    "state" : "RUNNING",
    "progress" : 10.0,
    "attributes" : { },
    "mr2AppInformation" : { }
  }, {
    "applicationId" : "application_1516618738289_0002",
    "name" : "Main",
    "startTime" : "1970-01-01T00:00:00.000Z",
    "user" : "maziyar",
    "pool" : "root.users.maziyar",
    "state" : "RUNNING",
    "progress" : 10.0,
    "attributes" : { },
    "mr2AppInformation" : { }
  } ],
  "warnings" : [ ]
}

The startTime is in 1970 for some reason! This date is really famouse in Unix:

 

"January 1, 1970 is the so called Unix epoch. It's the date where they started counting the Unix time. If you get this date as a return value, it usually means that the conversion of your date to the Unix timestamp returned a (near-) zero result. So the date conversion doesn't succeed"

 

So is it the backend of Cloudera Manager that has ` returns 0` or the MySQL conversion some where pass unsupported format .

Contributor
Posts: 26
Registered: ‎11-04-2016

Re: YARN Applications display wrong formatted duration

I just upgraded the entire cluster to 5.14 and the issue still remains:

 

CDH: 5.14

CM: 5.14

Announcements