Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

In yarn timeline service 2.0 not started

Highlighted

In yarn timeline service 2.0 not started

Explorer

stderr: /var/lib/ambari-agent/data/errors-2074.txt

Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/YARN/package/scripts/timelinereader.py", line 119, in <module>
ApplicationTimelineReader().execute()
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute
method(env)
File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/YARN/package/scripts/timelinereader.py", line 58, in start
hbase(action='start')
File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/YARN/package/scripts/hbase_service.py", line 80, in hbase
createTables()
File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/YARN/package/scripts/hbase_service.py", line 147, in createTables
logoutput=True)
File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
self.env.run()
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 263, in action_run
returns=self.resource.returns)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 72, in inner
result = function(command, **kwargs)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 102, in checked_call
tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy, returns=returns)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 150, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 308, in _call
raise ExecuteTimeoutException(err_msg)
resource_management.core.exceptions.ExecuteTimeoutException: Execution of 'ambari-sudo.sh su yarn-ats -l -s /bin/bash -c 'export PATH='"'"'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/var/lib/ambari-agent'"'"' ; sleep 10;export HBASE_CLASSPATH_PREFIX=/usr/hdp/3.1.0.0-78/hadoop-yarn/timelineservice/*; /usr/hdp/3.1.0.0-78/hbase/bin/hbase --config /usr/hdp/3.1.0.0-78/hadoop/conf/embedded-yarn-ats-hbase org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator -Dhbase.client.retries.number=35 -create -s'' was killed due timeout after 300 seconds

4 REPLIES 4

Re: In yarn timeline service 2.0 not started

Expert Contributor

@Manoj690  Can you check if you hit same error when you run this manually from cli -

 

Login to Ambari server node and execute below command -

 

$cd /var/lib/ambari-server/

$ambari-sudo.sh su yarn-ats -l -s /bin/bash -c 'export PATH='"'"'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/var/lib/ambari-agent'"'"' ; sleep 10;export HBASE_CLASSPATH_PREFIX=/usr/hdp/3.1.0.0-78/hadoop-yarn/timelineservice/*; /usr/hdp/3.1.0.0-78/hbase/bin/hbase --config /usr/hdp/3.1.0.0-78/hadoop/conf/embedded-yarn-ats-hbase org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator -Dhbase.client.retries.number=35 -create -s'

 

 

Re: In yarn timeline service 2.0 not started

Explorer
Can you check if you hit same error when you run this manually from cli -(*Yes
its get the same error*)


/var/lib/ambari-server# ambari-sudo.sh su yarn-ats -l -s /bin/bash -c
'export
PATH='"'"'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/var/lib/ambari-agent'"'"'
; sleep 10;export
HBASE_CLASSPATH_PREFIX=/usr/hdp/3.1.0.0-78/hadoop-yarn/timelineservice/*;
/usr/hdp/3.1.0.0-78/hbase/bin/hbase --config
/usr/hdp/3.1.0.0-78/hadoop/conf/embedded-yarn-ats-hbase
org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator
-Dhbase.client.retries.number=35 -create -s'
ambari-sudo.sh: command not found

Re: In yarn timeline service 2.0 not started

Mentor

@Manoj690 

Whenever you have an issue with an HDP component the first port of call is the /var/log/[$component] ATS is part of the YARN family so naturally, you should check the yarn logs under /var/log/hadoop-yarn/yarn/*  look for a missing conf/file system full or out of yarn memory etc

The most important file to check is highlighted in red


# cd /var/log/hadoop-yarn/yarn
# ls -al *.log
-rw-r--r-- 1 yarn hadoop 922643 Nov 21 19:18 hadoop-mapreduce.jobsummary.log
....
.....
-rw-r--r-- 1 yarn hadoop 91581388 Nov 21 21:25 hadoop-yarn-timelinereader-xxx.com.log
-rw-r--r-- 1 yarn hadoop 6738831 Nov 21 21:26 hadoop-yarn-timelineserver-xxx.com.log
-rw-r--r-- 1 yarn hadoop 62673 Nov 21 21:25 rm-audit.log

 

The ATS uses HBase as a backend database so make sure that HBase is running before you start YARN ats

Start by reading through this document how to clean up hdfs ATS data see the official documentation


Shutdown all YARN services !!!

Clean up zookeeper ATS data the example here is for insecure clusters, you will probably have another /atsv2-hbase-secure znode for kerberised clusters  use the  zookeeper CLI to remone the entry  rmr /atsv2-hbase-unsecure

$ /usr/hdp/3.1.0.0-78/zookeeper/bin/zkCli.sh
...
WATCHER::
....
WatchedEvent state:SyncConnected type:None path:null
[zk: localhost:2181(CONNECTED) 0] ls /atsv2-hbase-unsecure
[rs, splitWAL, backup-masters, table-lock, draining, master-maintenance, table]

Using the zookeeper CLI remove the ATSv2 znode if your cluster is kerberized you could see a different entry like /atsv2-hbase-secure

Delete the znode
[zk: localhost:2181(CONNECTED) 1] rmr /atsv2-hbase-unsecure

Restart the YARN services,
Restart ambari server to reload the new config.


Should you see stale configs restart those too including all services on the host where the ATS server is configured.

I assume this is not a production cluster else call Cloudera support but the solution follows the same path !!

HTH

Re: In yarn timeline service 2.0 not started

Mentor

@Manoj690 

 

Any updates on this thread?

Don't have an account?
Coming from Hortonworks? Activate your account here