Timeline Service V2.0 Reader not starting

[#1]

Everything in my HDP is working except Timeline Service V2.0 Reader. When I try to start it I get the following error log:

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/YARN/package/scripts/timelinereader.py", line 108, in <module>
    ApplicationTimelineReader().execute()
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 353, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/YARN/package/scripts/timelinereader.py", line 51, in start
    hbase(action='start')
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/YARN/package/scripts/hbase_service.py", line 80, in hbase
    createTables()
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/YARN/package/scripts/hbase_service.py", line 147, in createTables
    logoutput=True)
  File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
    self.env.run()
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 263, in action_run
    returns=self.resource.returns)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 72, in inner
    result = function(command, **kwargs)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 102, in checked_call
    tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy, returns=returns)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 150, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 308, in _call
    raise ExecuteTimeoutException(err_msg)
resource_management.core.exceptions.ExecuteTimeoutException: Execution of 'ambari-sudo.sh su yarn-ats -l -s /bin/bash -c 'export  PATH='"'"'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/local/texlive/2016/bin/x86_64-linux:/usr/local/texlive/2016/bin/x86_64-linux:/usr/local/texlive/2016/bin/x86_64-linux:/usr/lib64/qt-3.3/bin:/usr/local/texlive/2016/bin/x86_64-linux:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/maven/bin:/root/bin:/opt/maven/bin:/opt/maven/bin:/var/lib/ambari-agent'"'"' ; sleep 10;export HBASE_CLASSPATH_PREFIX=/usr/hdp/3.0.0.0-1634/hadoop-yarn/timelineservice/*; /usr/hdp/3.0.0.0-1634/hbase/bin/hbase --config /usr/hdp/3.0.0.0-1634/hadoop/conf/embedded-yarn-ats-hbase org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator -Dhbase.client.retries.number=35 -create -s'' was killed due timeout after 300 seconds

[#2]

@Daniel Zafar It looks like service restart is taking much longer time and Ambari is timing out - can you timeline service logs to understand why it is taking that long?

[#3]

Interestingly I don't have anything in the logs from today at all. I have tried to restart it today....

I am looking in:

/var/log/hadoop-yarn/yarn/hadoop-yarn-timelinereader-MYNODE01.log

Is that right?

[#4]

@Daniel Zafar yes - that is correct, its correct path. May be you can attempt to execute manually and see what happens.

 1. cd /var/lib/ambari-agent
 2.

./ambari-sudo.sh su yarn-ats -l -s /bin/bash -c 'export  PATH='"'"'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/local/texlive/2016/bin/x86_64-linux:/usr/local/texlive/2016/bin/x86_64-linux:/usr/local/texlive/2016/bin/x86_64-linux:/usr/lib64/qt-3.3/bin:/usr/local/texlive/2016/bin/x86_64-linux:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/maven/bin:/root/bin:/opt/maven/bin:/opt/maven/bin:/var/lib/ambari-agent'"'"' ; sleep 10;export HBASE_CLASSPATH_PREFIX=/usr/hdp/3.0.0.0-1634/hadoop-yarn/timelineservice/*; /usr/hdp/3.0.0.0-1634/hbase/bin/hbase --config /usr/hdp/3.0.0.0-1634/hadoop/conf/embedded-yarn-ats-hbase org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator -Dhbase.client.retries.number=35 -create -s'

This would print some logging in the UI

[#5]

@amarnath reddy pappu

Good idea! This is what I got:

[root@DRIVER_NODE ambari-agent]# ./ambari-sudo.sh su yarn-ats -l -s /bin/bash -c 'export PATH='"'"'/usr/sbin:/sbin:/usr/lib/ambari-server/:/usr/local/texlive/2016/bin/x86_64-linux:/usr/local/texlive/2016/bin/x86_64-linux:/usr/local/texlive/2016/bin/x86_64-linux:/usr/lib64/qt-3.3/bin:/usr/local/texlive/2016/bin/x86_64-linux:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/maven/bin:/root/bin:/opt/maven/bin:/opt/maven/bin:/var/lib/ambari-agent'"'"' ; sleep 10;export HBASE_CLASSPATH_PREFIX=/usr/hdp/3.0.0.0-1634/hadoop-yarn/timelineservice/; /usr/hdp/3.0.0.0-1634/hbase/bin/hbase --config /usr/hdp/3.0.0.0-1634/hadoop/conf/embedded-yarn-ats-hbase org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator -Dhbase.client.retries.number=35 -create -s'
Error: Could not find or load main class org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator

[#6]

After the error "Error:Could not find or load main class org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator", following the steps in below link fixed the issue.

Enable System Service Mode On an Upgraded Cluster

[#7]

Hi @Daniel zafar,

I looks like your package has some missing jars.

Can you please confirm whats the content of your folder ,this is how mine looks like.

If my assumption is correct yours will be missing : webapps, timelineservice and yarn-service-example folders.

 [root@anaik1 hadoop-yarn]# pwd
/usr/hdp/3.0.0.0-1634/hadoop-yarn
[root@anaik1 hadoop-yarn]# ls -l
total 12544
drwxr-xr-x. 2 root root      93 Jul 17 06:00 bin
lrwxrwxrwx. 1 root root      16 Jul 17 06:00 conf -> /etc/hadoop/conf
drwxr-xr-x. 3 root root      30 Jul 17 06:00 etc
-rw-r--r--. 1 root root 3097487 Jul 12 21:02 hadoop-yarn-api-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx. 1 root root      38 Jul 17 06:00 hadoop-yarn-api.jar -> hadoop-yarn-api-3.1.0.3.0.0.0-1634.jar
-rw-r--r--. 1 root root  110407 Jul 12 21:02 hadoop-yarn-applications-distributedshell-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx. 1 root root      64 Jul 17 06:00 hadoop-yarn-applications-distributedshell.jar -> hadoop-yarn-applications-distributedshell-3.1.0.3.0.0.0-1634.jar
-rw-r--r--. 1 root root   55399 Jul 12 21:02 hadoop-yarn-applications-unmanaged-am-launcher-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx. 1 root root      69 Jul 17 06:00 hadoop-yarn-applications-unmanaged-am-launcher.jar -> hadoop-yarn-applications-unmanaged-am-launcher-3.1.0.3.0.0.0-1634.jar
-rw-r--r--. 1 root root  300539 Jul 12 21:02 hadoop-yarn-client-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx. 1 root root      41 Jul 17 06:00 hadoop-yarn-client.jar -> hadoop-yarn-client-3.1.0.3.0.0.0-1634.jar
-rw-r--r--. 1 root root 2814263 Jul 12 21:02 hadoop-yarn-common-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx. 1 root root      41 Jul 17 06:00 hadoop-yarn-common.jar -> hadoop-yarn-common-3.1.0.3.0.0.0-1634.jar
-rw-r--r--. 1 root root  224598 Jul 12 21:02 hadoop-yarn-registry-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx. 1 root root      43 Jul 17 06:00 hadoop-yarn-registry.jar -> hadoop-yarn-registry-3.1.0.3.0.0.0-1634.jar
-rw-r--r--. 1 root root  294798 Jul 12 21:02 hadoop-yarn-server-applicationhistoryservice-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx. 1 root root      67 Jul 17 06:00 hadoop-yarn-server-applicationhistoryservice.jar -> hadoop-yarn-server-applicationhistoryservice-3.1.0.3.0.0.0-1634.jar
-rw-r--r--. 1 root root 1336803 Jul 12 21:02 hadoop-yarn-server-common-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx. 1 root root      48 Jul 17 06:00 hadoop-yarn-server-common.jar -> hadoop-yarn-server-common-3.1.0.3.0.0.0-1634.jar
-rw-r--r--. 1 root root 1293507 Jul 12 21:02 hadoop-yarn-server-nodemanager-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx. 1 root root      53 Jul 17 06:00 hadoop-yarn-server-nodemanager.jar -> hadoop-yarn-server-nodemanager-3.1.0.3.0.0.0-1634.jar
-rw-r--r--. 1 root root 2172705 Jul 12 21:02 hadoop-yarn-server-resourcemanager-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx. 1 root root      57 Jul 17 06:00 hadoop-yarn-server-resourcemanager.jar -> hadoop-yarn-server-resourcemanager-3.1.0.3.0.0.0-1634.jar
-rw-r--r--. 1 root root  149224 Jul 12 21:02 hadoop-yarn-server-router-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx. 1 root root      48 Jul 17 06:00 hadoop-yarn-server-router.jar -> hadoop-yarn-server-router-3.1.0.3.0.0.0-1634.jar
-rw-r--r--. 1 root root   94342 Jul 12 21:02 hadoop-yarn-server-sharedcachemanager-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx. 1 root root      60 Jul 17 06:00 hadoop-yarn-server-sharedcachemanager.jar -> hadoop-yarn-server-sharedcachemanager-3.1.0.3.0.0.0-1634.jar
-rw-r--r--. 1 root root   43865 Jul 12 21:02 hadoop-yarn-server-tests-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx. 1 root root      47 Jul 17 06:00 hadoop-yarn-server-tests.jar -> hadoop-yarn-server-tests-3.1.0.3.0.0.0-1634.jar
-rw-r--r--. 1 root root   93081 Jul 12 21:02 hadoop-yarn-server-timeline-pluginstorage-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx. 1 root root      64 Jul 17 06:00 hadoop-yarn-server-timeline-pluginstorage.jar -> hadoop-yarn-server-timeline-pluginstorage-3.1.0.3.0.0.0-1634.jar
-rw-r--r--. 1 root root   80404 Jul 12 21:02 hadoop-yarn-server-web-proxy-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx. 1 root root      51 Jul 17 06:00 hadoop-yarn-server-web-proxy.jar -> hadoop-yarn-server-web-proxy-3.1.0.3.0.0.0-1634.jar
-rw-r--r--. 1 root root  103378 Jul 12 21:02 hadoop-yarn-services-api-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx. 1 root root      47 Jul 17 06:00 hadoop-yarn-services-api.jar -> hadoop-yarn-services-api-3.1.0.3.0.0.0-1634.jar
-rw-r--r--. 1 root root  539627 Jul 12 21:02 hadoop-yarn-services-core-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx. 1 root root      48 Jul 17 06:00 hadoop-yarn-services-core.jar -> hadoop-yarn-services-core-3.1.0.3.0.0.0-1634.jar
drwxr-xr-x. 2 root root    4096 Jul 17 06:00 lib
drwxr-xr-x. 2 root root      49 Jul 17 06:00 sbin
drwxr-xr-x. 3 root root    4096 Jul 17 06:00 timelineservice
drwxr-xr-x. 3 root root      16 Jul 17 06:00 webapps
drwxr-xr-x. 5 root root      51 Jul 17 06:00 yarn-service-examples

[#8]

@Akhil S Naik

Hm, I actually have all of those, any other ideas?

<br>[root@NCIENSPK01 ~]# cd /usr/hdp/3.0.0.0-1634/hadoop-yarn
[root@NCIENSPK01 hadoop-yarn]# ls -l
total 12544
drwxr-xr-x 2 root root      93 Aug 10 17:51 bin
lrwxrwxrwx 1 root root      16 Aug 10 17:51 conf -> /etc/hadoop/conf
drwxr-xr-x 3 root root      30 Aug 10 17:51 etc
-rw-r--r-- 1 root root 3097487 Jul 12 16:02 hadoop-yarn-api-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx 1 root root      38 Aug 10 17:51 hadoop-yarn-api.jar -> hadoop-yarn-api-3.1.0.3.0.0.0-1634.jar
-rw-r--r-- 1 root root  110407 Jul 12 16:02 hadoop-yarn-applications-distributedshell-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx 1 root root      64 Aug 10 17:51 hadoop-yarn-applications-distributedshell.jar -> hadoop-yarn-applications-distributedshell-3.1.0.3.0.0.0-1634.jar
-rw-r--r-- 1 root root   55399 Jul 12 16:02 hadoop-yarn-applications-unmanaged-am-launcher-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx 1 root root      69 Aug 10 17:51 hadoop-yarn-applications-unmanaged-am-launcher.jar -> hadoop-yarn-applications-unmanaged-am-launcher-3.1.0.3.0.0.0-1634.jar
-rw-r--r-- 1 root root  300539 Jul 12 16:02 hadoop-yarn-client-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx 1 root root      41 Aug 10 17:51 hadoop-yarn-client.jar -> hadoop-yarn-client-3.1.0.3.0.0.0-1634.jar
-rw-r--r-- 1 root root 2814263 Jul 12 16:02 hadoop-yarn-common-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx 1 root root      41 Aug 10 17:51 hadoop-yarn-common.jar -> hadoop-yarn-common-3.1.0.3.0.0.0-1634.jar
-rw-r--r-- 1 root root  224598 Jul 12 16:02 hadoop-yarn-registry-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx 1 root root      43 Aug 10 17:51 hadoop-yarn-registry.jar -> hadoop-yarn-registry-3.1.0.3.0.0.0-1634.jar
-rw-r--r-- 1 root root  294798 Jul 12 16:02 hadoop-yarn-server-applicationhistoryservice-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx 1 root root      67 Aug 10 17:51 hadoop-yarn-server-applicationhistoryservice.jar -> hadoop-yarn-server-applicationhistoryservice-3.1.0.3.0.0.0-1634.jar
-rw-r--r-- 1 root root 1336803 Jul 12 16:02 hadoop-yarn-server-common-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx 1 root root      48 Aug 10 17:51 hadoop-yarn-server-common.jar -> hadoop-yarn-server-common-3.1.0.3.0.0.0-1634.jar
-rw-r--r-- 1 root root 1293507 Jul 12 16:02 hadoop-yarn-server-nodemanager-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx 1 root root      53 Aug 10 17:51 hadoop-yarn-server-nodemanager.jar -> hadoop-yarn-server-nodemanager-3.1.0.3.0.0.0-1634.jar
-rw-r--r-- 1 root root 2172705 Jul 12 16:02 hadoop-yarn-server-resourcemanager-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx 1 root root      57 Aug 10 17:51 hadoop-yarn-server-resourcemanager.jar -> hadoop-yarn-server-resourcemanager-3.1.0.3.0.0.0-1634.jar
-rw-r--r-- 1 root root  149224 Jul 12 16:02 hadoop-yarn-server-router-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx 1 root root      48 Aug 10 17:51 hadoop-yarn-server-router.jar -> hadoop-yarn-server-router-3.1.0.3.0.0.0-1634.jar
-rw-r--r-- 1 root root   94342 Jul 12 16:02 hadoop-yarn-server-sharedcachemanager-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx 1 root root      60 Aug 10 17:51 hadoop-yarn-server-sharedcachemanager.jar -> hadoop-yarn-server-sharedcachemanager-3.1.0.3.0.0.0-1634.jar
-rw-r--r-- 1 root root   43865 Jul 12 16:02 hadoop-yarn-server-tests-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx 1 root root      47 Aug 10 17:51 hadoop-yarn-server-tests.jar -> hadoop-yarn-server-tests-3.1.0.3.0.0.0-1634.jar
-rw-r--r-- 1 root root   93081 Jul 12 16:02 hadoop-yarn-server-timeline-pluginstorage-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx 1 root root      64 Aug 10 17:51 hadoop-yarn-server-timeline-pluginstorage.jar -> hadoop-yarn-server-timeline-pluginstorage-3.1.0.3.0.0.0-1634.jar
-rw-r--r-- 1 root root   80404 Jul 12 16:02 hadoop-yarn-server-web-proxy-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx 1 root root      51 Aug 10 17:51 hadoop-yarn-server-web-proxy.jar -> hadoop-yarn-server-web-proxy-3.1.0.3.0.0.0-1634.jar
-rw-r--r-- 1 root root  103378 Jul 12 16:02 hadoop-yarn-services-api-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx 1 root root      47 Aug 10 17:51 hadoop-yarn-services-api.jar -> hadoop-yarn-services-api-3.1.0.3.0.0.0-1634.jar
-rw-r--r-- 1 root root  539627 Jul 12 16:02 hadoop-yarn-services-core-3.1.0.3.0.0.0-1634.jar
lrwxrwxrwx 1 root root      48 Aug 10 17:51 hadoop-yarn-services-core.jar -> hadoop-yarn-services-core-3.1.0.3.0.0.0-1634.jar
drwxr-xr-x 2 root root    4096 Aug 10 17:51 lib
drwxr-xr-x 2 root root      49 Aug 10 17:51 sbin
drwxr-xr-x 3 root root    4096 Aug 10 17:51 timelineservice
drwxr-xr-x 3 root root      16 Aug 10 17:51 webapps
drwxr-xr-x 5 root root      51 Aug 10 17:51 yarn-service-examples

Then in timeline service:

[root@NCIENSPK01 hadoop-yarn]# ls timelineservice
hadoop-yarn-server-timelineservice-3.1.0.3.0.0.0-1634.jar               hadoop-yarn-server-timelineservice-hbase-common-3.1.0.3.0.0.0-1634.jar       hadoop-yarn-server-timelineservice-hbase-coprocessor.jar
hadoop-yarn-server-timelineservice-hbase-client-3.1.0.3.0.0.0-1634.jar  hadoop-yarn-server-timelineservice-hbase-common.jar                          hadoop-yarn-server-timelineservice.jar
hadoop-yarn-server-timelineservice-hbase-client.jar  

[#9]

@Daniel Zafar

Hi,

I'm also facing the same error, were you able to find anything?

# ./ambari-sudo.sh su yarn-ats -l -s /bin/bash -c 'export PATH='"'"'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/local/texlive/2016/bin/x86_64-linux:/usr/local/texlive/2016/bin/x86_64-linux:/usr/local/texlive/2016/bin/x86_64-linux:/usr/lib64/qt-3.3/bin:/usr/local/texlive/2016/bin/x86_64-linux:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/maven/bin:/root/bin:/opt/maven/bin:/opt/maven/bin:/var/lib/ambari-agent'"'"' ; sleep 10;export HBASE_CLASSPATH_PREFIX=/usr/hdp/3.0.1.0-187/hadoop-yarn/timelineservice/^*; /usr/hdp/3.0.1.0-187/hbase/bin/hbase --config /usr/hdp/3.0.1.0-187//hadoop/conf/embedded-yarn-ats-hbase org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator -Dhbase.client.retries.number=35 -create -s' 
Error: Could not find or load main class org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator

[#10 · Accepted solution]

Hi @Daniel Zafar,

Can you please navigate to the host where Timeline Reader is installed and Install Hbase Client in that host and let me know the result.

Seeing the code base(the missing file is in hadoop-yarn-server-timelineservice-hbase-client) I have strong feeling this can fix the issue.

Code reference : https://github.com/hortonworks/hadoop-release/blob/HDP-3.0.0.0-1634-tag/hadoop-yarn-project/hadoop-y...

Please log in and accept the answer if you find it helpful 🙂

[#11]

@Akhil S Naik That worked! Nice job!

[#12]

Hello @Akhil S Naik, @Daniel Zafar

I am running on HDP-3.0.1 and Yarn 3.1.0 and ran into the same issue which says resource_management.core.exceptions.ExecuteTimeoutException: Execution of 'ambari-sudo.sh su yarn-ats -l -s /bin/bash -c 'export PATH='"'"'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/var/lib/ambari-agent:/var/lib/ambari-agent'"'"' ; sleep 10;export HBASE_CLASSPATH_PREFIX=/usr/hdp/3.0.1.0-187/hadoop-yarn/timelineservice/*; /usr/hdp/3.0.1.0-187/hbase/bin/hbase --config /usr/hdp/3.0.1.0-187/hadoop/conf/embedded-yarn-ats-hbase org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator -Dhbase.client.retries.number=35 -create -s'' was killed due timeout after 300 seconds

Also can you please specify how to install hbase client here? Is it as a part of hbase service through ambari or just the hbase client using the cli? Any guidance here please.

[#13]

Was running into this exact same problem. Here is how I installed HBase Client from via the Ambari UI...

1. In the Ambari UI, go to Hosts then click the host you want to install the hbase client component on

2. In the list on components, you will have option to add more, see...

3. From here I installed the HBase client

4. Then stopped and restarted the cluster via Ambari UI (got notification of stale configs (though not sure if this was my problem all along))

One thing that was weird is that I did not change any configs or install anything new on the host nodes between trying to restart and running into this error and up until now everything appeared to be working fine. @Akhil S Naik, is there any reason that you could think of why this would only be happening now?

[#14]

Myself I do not have any issues with class not found but:

client.ConnectionImplementation: Retrieve cluster id failed

may be any related so posting

018-10-19 22:48:33,152 INFO  [ReadOnlyZKClient-emltgh01.emtst.lpemrz.com:2181@0x38102d01] zookeeper.ZooKeeper: Client environment:java.library.path=:/usr/hdp/3.0.1.0-187/hadoop/lib/native/Linux-amd64-64:/usr/hdp/3.0.1.0-187/hadoop/lib/native
2018-10-19 22:48:33,152 INFO  [ReadOnlyZKClient-emltgh01.emtst.lpemrz.com:2181@0x38102d01] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
2018-10-19 22:48:33,152 INFO  [ReadOnlyZKClient-emltgh01.emtst.lpemrz.com:2181@0x38102d01] zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
2018-10-19 22:48:33,152 INFO  [ReadOnlyZKClient-emltgh01.emtst.lpemrz.com:2181@0x38102d01] zookeeper.ZooKeeper: Client environment:os.name=Linux
2018-10-19 22:48:33,152 INFO  [ReadOnlyZKClient-emltgh01.emtst.lpemrz.com:2181@0x38102d01] zookeeper.ZooKeeper: Client environment:os.arch=amd64
2018-10-19 22:48:33,152 INFO  [ReadOnlyZKClient-emltgh01.emtst.lpemrz.com:2181@0x38102d01] zookeeper.ZooKeeper: Client environment:os.version=3.10.0-514.21.1.el7.x86_64
2018-10-19 22:48:33,152 INFO  [ReadOnlyZKClient-emltgh01.emtst.lpemrz.com:2181@0x38102d01] zookeeper.ZooKeeper: Client environment:user.name=yarn-ats
2018-10-19 22:48:33,152 INFO  [ReadOnlyZKClient-emltgh01.emtst.lpemrz.com:2181@0x38102d01] zookeeper.ZooKeeper: Client environment:user.home=/home/yarn-ats
2018-10-19 22:48:33,152 INFO  [ReadOnlyZKClient-emltgh01.emtst.lpemrz.com:2181@0x38102d01] zookeeper.ZooKeeper: Client environment:user.dir=/home/yarn-ats
2018-10-19 22:48:33,154 INFO  [ReadOnlyZKClient-emltgh01.emtst.lpemrz.com:2181@0x38102d01] zookeeper.ZooKeeper: Initiating client connection, connectString=emltgh01.emtst.lpemrz.com:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$13/966280619@62068d2d
2018-10-19 22:48:33,171 INFO  [ReadOnlyZKClient-emltgh01.emtst.lpemrz.com:2181@0x38102d01-SendThread(emltgh01.emtst.lpemrz.com:2181)] zookeeper.ClientCnxn: Opening socket connection to server emltgh01.emtst.lpemrz.com/10.10.13.100:2181. Will not attempt to authenticate using SASL (unknown error)
2018-10-19 22:48:33,174 INFO  [ReadOnlyZKClient-emltgh01.emtst.lpemrz.com:2181@0x38102d01-SendThread(emltgh01.emtst.lpemrz.com:2181)] zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.10.13.100:38167, server: emltgh01.emtst.lpemrz.com/10.10.13.100:2181
2018-10-19 22:48:33,179 INFO  [ReadOnlyZKClient-emltgh01.emtst.lpemrz.com:2181@0x38102d01-SendThread(emltgh01.emtst.lpemrz.com:2181)] zookeeper.ClientCnxn: Session establishment complete on server emltgh01.emtst.lpemrz.com/10.10.13.100:2181, sessionid = 0x1668d9e70b70055, negotiated timeout = 40000
2018-10-19 22:48:33,188 WARN  [main] client.ConnectionImplementation: Retrieve cluster id failed
java.util.concurrent.ExecutionException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /atsv2-hbase-unsecure/hbaseid
	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
	at org.apache.hadoop.hbase.client.ConnectionImplementation.retrieveClusterId(ConnectionImplementation.java:527)
	at org.apache.hadoop.hbase.client.ConnectionImplementation.<init>(ConnectionImplementation.java:287)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:219)
	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:114)
	at org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator.createAllTables(TimelineSchemaCreator.java:301)
	at org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator.createAllSchemas(TimelineSchemaCreator.java:277)
	at org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator.main(TimelineSchemaCreator.java:146)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /atsv2-hbase-unsecure/hbaseid
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:168)
	at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:323)
	at java.lang.Thread.run(Thread.java:745)

[#15]

@Marek Martofel Following the steps in below link fixed this issue:

Enable System Service Mode On an Upgraded Cluster

[#16]

Thanks CIBI, It is worked for me..

[#17]

@ccibi75Thanks for the solution to resolve timeline server v2.0 start issue in HDP3.x. It worked!!!

[#18]

Thanks, ccibi75

This worked for me.

[#19]

Many thanks Cibi! It works now. Indeed yarn-system queue was with 0% capacity and is_hbase_system_service_launch was false.