Timeline Service V2.0 Reader not starting

[#1]

Everything in my HDP is working except Timeline Service V2.0 Reader. When I try to start it I get the following error log:

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/YARN/package/scripts/timelinereader.py", line 108, in <module>
    ApplicationTimelineReader().execute()
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 353, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/YARN/package/scripts/timelinereader.py", line 51, in start
    hbase(action='start')
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/YARN/package/scripts/hbase_service.py", line 80, in hbase
    createTables()
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/YARN/package/scripts/hbase_service.py", line 147, in createTables
    logoutput=True)
  File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
    self.env.run()
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 263, in action_run
    returns=self.resource.returns)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 72, in inner
    result = function(command, **kwargs)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 102, in checked_call
    tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy, returns=returns)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 150, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 308, in _call
    raise ExecuteTimeoutException(err_msg)
resource_management.core.exceptions.ExecuteTimeoutException: Execution of 'ambari-sudo.sh su yarn-ats -l -s /bin/bash -c 'export  PATH='"'"'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/local/texlive/2016/bin/x86_64-linux:/usr/local/texlive/2016/bin/x86_64-linux:/usr/local/texlive/2016/bin/x86_64-linux:/usr/lib64/qt-3.3/bin:/usr/local/texlive/2016/bin/x86_64-linux:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/maven/bin:/root/bin:/opt/maven/bin:/opt/maven/bin:/var/lib/ambari-agent'"'"' ; sleep 10;export HBASE_CLASSPATH_PREFIX=/usr/hdp/3.0.0.0-1634/hadoop-yarn/timelineservice/*; /usr/hdp/3.0.0.0-1634/hbase/bin/hbase --config /usr/hdp/3.0.0.0-1634/hadoop/conf/embedded-yarn-ats-hbase org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator -Dhbase.client.retries.number=35 -create -s'' was killed due timeout after 300 seconds

[#2]

@Daniel Zafar It looks like service restart is taking much longer time and Ambari is timing out - can you timeline service logs to understand why it is taking that long?

[#3]

Interestingly I don't have anything in the logs from today at all. I have tried to restart it today....

I am looking in:

/var/log/hadoop-yarn/yarn/hadoop-yarn-timelinereader-MYNODE01.log

Is that right?

[#4]

@Daniel Zafar yes - that is correct, its correct path. May be you can attempt to execute manually and see what happens.

 1. cd /var/lib/ambari-agent
 2.

./ambari-sudo.sh su yarn-ats -l -s /bin/bash -c 'export  PATH='"'"'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/local/texlive/2016/bin/x86_64-linux:/usr/local/texlive/2016/bin/x86_64-linux:/usr/local/texlive/2016/bin/x86_64-linux:/usr/lib64/qt-3.3/bin:/usr/local/texlive/2016/bin/x86_64-linux:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/maven/bin:/root/bin:/opt/maven/bin:/opt/maven/bin:/var/lib/ambari-agent'"'"' ; sleep 10;export HBASE_CLASSPATH_PREFIX=/usr/hdp/3.0.0.0-1634/hadoop-yarn/timelineservice/*; /usr/hdp/3.0.0.0-1634/hbase/bin/hbase --config /usr/hdp/3.0.0.0-1634/hadoop/conf/embedded-yarn-ats-hbase org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator -Dhbase.client.retries.number=35 -create -s'

This would print some logging in the UI

[#5]

@amarnath reddy pappu

Good idea! This is what I got:

[root@DRIVER_NODE ambari-agent]# ./ambari-sudo.sh su yarn-ats -l -s /bin/bash -c 'export PATH='"'"'/usr/sbin:/sbin:/usr/lib/ambari-server/:/usr/local/texlive/2016/bin/x86_64-linux:/usr/local/texlive/2016/bin/x86_64-linux:/usr/local/texlive/2016/bin/x86_64-linux:/usr/lib64/qt-3.3/bin:/usr/local/texlive/2016/bin/x86_64-linux:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/maven/bin:/root/bin:/opt/maven/bin:/opt/maven/bin:/var/lib/ambari-agent'"'"' ; sleep 10;export HBASE_CLASSPATH_PREFIX=/usr/hdp/3.0.0.0-1634/hadoop-yarn/timelineservice/; /usr/hdp/3.0.0.0-1634/hbase/bin/hbase --config /usr/hdp/3.0.0.0-1634/hadoop/conf/embedded-yarn-ats-hbase org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator -Dhbase.client.retries.number=35 -create -s'
Error: Could not find or load main class org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator

[#6]

After the error "Error:Could not find or load main class org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator", following the steps in below link fixed the issue.

Enable System Service Mode On an Upgraded Cluster

[#7]

Hi @Daniel zafar,

I looks like your package has some missing jars.

Can you please confirm whats the content of your folder ,this is how mine looks like.

If my assumption is correct yours will be missing : webapps, timelineservice and yarn-service-example folders.

 [root@anaik1 hadoop-yarn]# pwd
/usr/hdp/3.0.0.0-1634/hadoop-yarn
[root@anaik1 hadoop-yarn]# ls -l
total 12544
drwxr-xr-x. 2 root root      93 Jul 17 06:00 bin
lrwxrwxrwx. 1 root root      16 Jul 17 06:00 conf -> /etc/hadoop/conf
drwxr-xr-x. 3 root root      30 Jul 17 06:00 etc
-rw-r--r--. 1 root root 3097487 Jul 12 21:02 hadoop-yarn-api-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx. 1 root root      38 Jul 17 06:00 hadoop-yarn-api.jar -> hadoop-yarn-api-3.1.0.3.0.0.0-1634.jar
-rw-r--r--. 1 root root  110407 Jul 12 21:02 hadoop-yarn-applications-distributedshell-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx. 1 root root      64 Jul 17 06:00 hadoop-yarn-applications-distributedshell.jar -> hadoop-yarn-applications-distributedshell-3.1.0.3.0.0.0-1634.jar
-rw-r--r--. 1 root root   55399 Jul 12 21:02 hadoop-yarn-applications-unmanaged-am-launcher-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx. 1 root root      69 Jul 17 06:00 hadoop-yarn-applications-unmanaged-am-launcher.jar -> hadoop-yarn-applications-unmanaged-am-launcher-3.1.0.3.0.0.0-1634.jar
-rw-r--r--. 1 root root  300539 Jul 12 21:02 hadoop-yarn-client-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx. 1 root root      41 Jul 17 06:00 hadoop-yarn-client.jar -> hadoop-yarn-client-3.1.0.3.0.0.0-1634.jar
-rw-r--r--. 1 root root 2814263 Jul 12 21:02 hadoop-yarn-common-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx. 1 root root      41 Jul 17 06:00 hadoop-yarn-common.jar -> hadoop-yarn-common-3.1.0.3.0.0.0-1634.jar
-rw-r--r--. 1 root root  224598 Jul 12 21:02 hadoop-yarn-registry-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx. 1 root root      43 Jul 17 06:00 hadoop-yarn-registry.jar -> hadoop-yarn-registry-3.1.0.3.0.0.0-1634.jar
-rw-r--r--. 1 root root  294798 Jul 12 21:02 hadoop-yarn-server-applicationhistoryservice-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx. 1 root root      67 Jul 17 06:00 hadoop-yarn-server-applicationhistoryservice.jar -> hadoop-yarn-server-applicationhistoryservice-3.1.0.3.0.0.0-1634.jar
-rw-r--r--. 1 root root 1336803 Jul 12 21:02 hadoop-yarn-server-common-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx. 1 root root      48 Jul 17 06:00 hadoop-yarn-server-common.jar -> hadoop-yarn-server-common-3.1.0.3.0.0.0-1634.jar
-rw-r--r--. 1 root root 1293507 Jul 12 21:02 hadoop-yarn-server-nodemanager-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx. 1 root root      53 Jul 17 06:00 hadoop-yarn-server-nodemanager.jar -> hadoop-yarn-server-nodemanager-3.1.0.3.0.0.0-1634.jar
-rw-r--r--. 1 root root 2172705 Jul 12 21:02 hadoop-yarn-server-resourcemanager-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx. 1 root root      57 Jul 17 06:00 hadoop-yarn-server-resourcemanager.jar -> hadoop-yarn-server-resourcemanager-3.1.0.3.0.0.0-1634.jar
-rw-r--r--. 1 root root  149224 Jul 12 21:02 hadoop-yarn-server-router-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx. 1 root root      48 Jul 17 06:00 hadoop-yarn-server-router.jar -> hadoop-yarn-server-router-3.1.0.3.0.0.0-1634.jar
-rw-r--r--. 1 root root   94342 Jul 12 21:02 hadoop-yarn-server-sharedcachemanager-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx. 1 root root      60 Jul 17 06:00 hadoop-yarn-server-sharedcachemanager.jar -> hadoop-yarn-server-sharedcachemanager-3.1.0.3.0.0.0-1634.jar
-rw-r--r--. 1 root root   43865 Jul 12 21:02 hadoop-yarn-server-tests-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx. 1 root root      47 Jul 17 06:00 hadoop-yarn-server-tests.jar -> hadoop-yarn-server-tests-3.1.0.3.0.0.0-1634.jar
-rw-r--r--. 1 root root   93081 Jul 12 21:02 hadoop-yarn-server-timeline-pluginstorage-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx. 1 root root      64 Jul 17 06:00 hadoop-yarn-server-timeline-pluginstorage.jar -> hadoop-yarn-server-timeline-pluginstorage-3.1.0.3.0.0.0-1634.jar
-rw-r--r--. 1 root root   80404 Jul 12 21:02 hadoop-yarn-server-web-proxy-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx. 1 root root      51 Jul 17 06:00 hadoop-yarn-server-web-proxy.jar -> hadoop-yarn-server-web-proxy-3.1.0.3.0.0.0-1634.jar
-rw-r--r--. 1 root root  103378 Jul 12 21:02 hadoop-yarn-services-api-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx. 1 root root      47 Jul 17 06:00 hadoop-yarn-services-api.jar -> hadoop-yarn-services-api-3.1.0.3.0.0.0-1634.jar
-rw-r--r--. 1 root root  539627 Jul 12 21:02 hadoop-yarn-services-core-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx. 1 root root      48 Jul 17 06:00 hadoop-yarn-services-core.jar -> hadoop-yarn-services-core-3.1.0.3.0.0.0-1634.jar
drwxr-xr-x. 2 root root    4096 Jul 17 06:00 lib
drwxr-xr-x. 2 root root      49 Jul 17 06:00 sbin
drwxr-xr-x. 3 root root    4096 Jul 17 06:00 timelineservice
drwxr-xr-x. 3 root root      16 Jul 17 06:00 webapps
drwxr-xr-x. 5 root root      51 Jul 17 06:00 yarn-service-examples

[#8]

@Akhil S Naik

Hm, I actually have all of those, any other ideas?

<br>[root@NCIENSPK01 ~]# cd /usr/hdp/3.0.0.0-1634/hadoop-yarn
[root@NCIENSPK01 hadoop-yarn]# ls -l
total 12544
drwxr-xr-x 2 root root      93 Aug 10 17:51 bin
lrwxrwxrwx 1 root root      16 Aug 10 17:51 conf -> /etc/hadoop/conf
drwxr-xr-x 3 root root      30 Aug 10 17:51 etc
-rw-r--r-- 1 root root 3097487 Jul 12 16:02 hadoop-yarn-api-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx 1 root root      38 Aug 10 17:51 hadoop-yarn-api.jar -> hadoop-yarn-api-3.1.0.3.0.0.0-1634.jar
-rw-r--r-- 1 root root  110407 Jul 12 16:02 hadoop-yarn-applications-distributedshell-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx 1 root root      64 Aug 10 17:51 hadoop-yarn-applications-distributedshell.jar -> hadoop-yarn-applications-distributedshell-3.1.0.3.0.0.0-1634.jar
-rw-r--r-- 1 root root   55399 Jul 12 16:02 hadoop-yarn-applications-unmanaged-am-launcher-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx 1 root root      69 Aug 10 17:51 hadoop-yarn-applications-unmanaged-am-launcher.jar -> hadoop-yarn-applications-unmanaged-am-launcher-3.1.0.3.0.0.0-1634.jar
-rw-r--r-- 1 root root  300539 Jul 12 16:02 hadoop-yarn-client-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx 1 root root      41 Aug 10 17:51 hadoop-yarn-client.jar -> hadoop-yarn-client-3.1.0.3.0.0.0-1634.jar
-rw-r--r-- 1 root root 2814263 Jul 12 16:02 hadoop-yarn-common-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx 1 root root      41 Aug 10 17:51 hadoop-yarn-common.jar -> hadoop-yarn-common-3.1.0.3.0.0.0-1634.jar
-rw-r--r-- 1 root root  224598 Jul 12 16:02 hadoop-yarn-registry-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx 1 root root      43 Aug 10 17:51 hadoop-yarn-registry.jar -> hadoop-yarn-registry-3.1.0.3.0.0.0-1634.jar
-rw-r--r-- 1 root root  294798 Jul 12 16:02 hadoop-yarn-server-applicationhistoryservice-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx 1 root root      67 Aug 10 17:51 hadoop-yarn-server-applicationhistoryservice.jar -> hadoop-yarn-server-applicationhistoryservice-3.1.0.3.0.0.0-1634.jar
-rw-r--r-- 1 root root 1336803 Jul 12 16:02 hadoop-yarn-server-common-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx 1 root root      48 Aug 10 17:51 hadoop-yarn-server-common.jar -> hadoop-yarn-server-common-3.1.0.3.0.0.0-1634.jar
-rw-r--r-- 1 root root 1293507 Jul 12 16:02 hadoop-yarn-server-nodemanager-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx 1 root root      53 Aug 10 17:51 hadoop-yarn-server-nodemanager.jar -> hadoop-yarn-server-nodemanager-3.1.0.3.0.0.0-1634.jar
-rw-r--r-- 1 root root 2172705 Jul 12 16:02 hadoop-yarn-server-resourcemanager-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx 1 root root      57 Aug 10 17:51 hadoop-yarn-server-resourcemanager.jar -> hadoop-yarn-server-resourcemanager-3.1.0.3.0.0.0-1634.jar
-rw-r--r-- 1 root root  149224 Jul 12 16:02 hadoop-yarn-server-router-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx 1 root root      48 Aug 10 17:51 hadoop-yarn-server-router.jar -> hadoop-yarn-server-router-3.1.0.3.0.0.0-1634.jar
-rw-r--r-- 1 root root   94342 Jul 12 16:02 hadoop-yarn-server-sharedcachemanager-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx 1 root root      60 Aug 10 17:51 hadoop-yarn-server-sharedcachemanager.jar -> hadoop-yarn-server-sharedcachemanager-3.1.0.3.0.0.0-1634.jar
-rw-r--r-- 1 root root   43865 Jul 12 16:02 hadoop-yarn-server-tests-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx 1 root root      47 Aug 10 17:51 hadoop-yarn-server-tests.jar -> hadoop-yarn-server-tests-3.1.0.3.0.0.0-1634.jar
-rw-r--r-- 1 root root   93081 Jul 12 16:02 hadoop-yarn-server-timeline-pluginstorage-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx 1 root root      64 Aug 10 17:51 hadoop-yarn-server-timeline-pluginstorage.jar -> hadoop-yarn-server-timeline-pluginstorage-3.1.0.3.0.0.0-1634.jar
-rw-r--r-- 1 root root   80404 Jul 12 16:02 hadoop-yarn-server-web-proxy-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx 1 root root      51 Aug 10 17:51 hadoop-yarn-server-web-proxy.jar -> hadoop-yarn-server-web-proxy-3.1.0.3.0.0.0-1634.jar
-rw-r--r-- 1 root root  103378 Jul 12 16:02 hadoop-yarn-services-api-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx 1 root root      47 Aug 10 17:51 hadoop-yarn-services-api.jar -> hadoop-yarn-services-api-3.1.0.3.0.0.0-1634.jar
-rw-r--r-- 1 root root  539627 Jul 12 16:02 hadoop-yarn-services-core-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx 1 root root      48 Aug 10 17:51 hadoop-yarn-services-core.jar -> hadoop-yarn-services-core-3.1.0.3.0.0.0-1634.jar
drwxr-xr-x 2 root root    4096 Aug 10 17:51 lib
drwxr-xr-x 2 root root      49 Aug 10 17:51 sbin
drwxr-xr-x 3 root root    4096 Aug 10 17:51 timelineservice
drwxr-xr-x 3 root root      16 Aug 10 17:51 webapps
drwxr-xr-x 5 root root      51 Aug 10 17:51 yarn-service-examples

Then in timeline service:

[root@NCIENSPK01 hadoop-yarn]# ls timelineservice
hadoop-yarn-server-timelineservice-3.1.0.3.0.0.0-1634.jar               hadoop-yarn-server-timelineservice-hbase-common-3.1.0.3.0.0.0-1634.jar       hadoop-yarn-server-timelineservice-hbase-coprocessor.jar
hadoop-yarn-server-timelineservice-hbase-client-3.1.0.3.0.0.0-1634.jar  hadoop-yarn-server-timelineservice-hbase-common.jar                          hadoop-yarn-server-timelineservice.jar
hadoop-yarn-server-timelineservice-hbase-client.jar  

[#9]

@Daniel Zafar

Hi,

I'm also facing the same error, were you able to find anything?

# ./ambari-sudo.sh su yarn-ats -l -s /bin/bash -c 'export PATH='"'"'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/local/texlive/2016/bin/x86_64-linux:/usr/local/texlive/2016/bin/x86_64-linux:/usr/local/texlive/2016/bin/x86_64-linux:/usr/lib64/qt-3.3/bin:/usr/local/texlive/2016/bin/x86_64-linux:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/maven/bin:/root/bin:/opt/maven/bin:/opt/maven/bin:/var/lib/ambari-agent'"'"' ; sleep 10;export HBASE_CLASSPATH_PREFIX=/usr/hdp/3.0.1.0-187/hadoop-yarn/timelineservice/^*; /usr/hdp/3.0.1.0-187/hbase/bin/hbase --config /usr/hdp/3.0.1.0-187//hadoop/conf/embedded-yarn-ats-hbase org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator -Dhbase.client.retries.number=35 -create -s' 
Error: Could not find or load main class org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator

[#10 · Accepted solution]

Hi @Daniel Zafar,

Can you please navigate to the host where Timeline Reader is installed and Install Hbase Client in that host and let me know the result.

Seeing the code base(the missing file is in hadoop-yarn-server-timelineservice-hbase-client) I have strong feeling this can fix the issue.

Code reference : https://github.com/hortonworks/hadoop-release/blob/HDP-3.0.0.0-1634-tag/hadoop-yarn-project/hadoop-y...

Please log in and accept the answer if you find it helpful 🙂