question Re: after upgrade hdp. in Archives of Support Questions (Read Only)

after upgrade hdp.

ashneesharma88 — Mon, 18 Jul 2016 20:39:02 GMT

I am getting error after upgrade the hdp version from 2.3.2.0-2950 to 2.3.4.0-3485.

HBASE_CLIENT in invalid state. Invalid transition. Invalid event: HOST_SVCCOMP_OP_IN_PROGRESS at INSTALL_FAILED hdp.

I tried

Re: after upgrade hdp.

mjohnson — Mon, 18 Jul 2016 20:57:11 GMT

Can you run the command '/usr/bin/hdp-select' from the command line ? and verify the correct version of HBASE is currently active.

The installation errors will be documented in the /var/log/ambari-agent/ambari-agent.log file. Any activity related to the HBASE_CLIENT installation can be identified by searching on each server for that string. Based on the message it seems like there is an installation state problem with one of the servers. To get a better idea of which server the problem exists and a better idea of the actual problem, you will want to check the amber-agent.log file for each server and execute the following grep command from the /var/log/ambari-agent directory: "grep HBASE_CLIENT amber-agent.log | grep INSTALLED". The servers which do not have an INSTALLED is probably the server where there was a problem, and you should find the error by just searching the ambari-agent.log file for the string "HBASE_CLIENT.

Re: after upgrade hdp.

ashneesharma88 — Mon, 18 Jul 2016 21:38:26 GMT

@mjohnson

this is failed with exit1:-

INFO 2016-07-18 09:51:22,789 PythonExecutor.py:114 - Command ['/usr/bin/python2.7', u'/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/hook.py', u'INSTALL', '/var/lib/ambari-agent/data/command-353.json', u'/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/after-INSTALL', '/var/lib/ambari-agent/data/structured-out-353.json', 'INFO', '/var/lib/ambari-agent/data/tmp'] failed with exitcode=1

Re: after upgrade hdp.

ashneesharma88 — Mon, 18 Jul 2016 21:42:06 GMT

@mjohnson

My problem is that ambari detect that the client installed, but when i am restarting the hbase client config it's getting failed.

Below is the output of hdp-select.

[hdfs@qacluster ~]$ /usr/bin/hdp-select accumulo-client - None accumulo-gc - None accumulo-master - None accumulo-monitor - None accumulo-tablet - None accumulo-tracer - None atlas-server - None falcon-client - None falcon-server - None flume-server - None hadoop-client - 2.3.4.0-3485 hadoop-hdfs-datanode - 2.3.4.0-3485 hadoop-hdfs-journalnode - 2.3.4.0-3485 hadoop-hdfs-namenode - 2.3.4.0-3485 hadoop-hdfs-nfs3 - 2.3.4.0-3485 hadoop-hdfs-portmap - 2.3.4.0-3485 hadoop-hdfs-secondarynamenode - 2.3.4.0-3485 hadoop-httpfs - 2.3.4.0-3485 hadoop-mapreduce-historyserver - 2.3.4.0-3485 hadoop-yarn-nodemanager - 2.3.4.0-3485 hadoop-yarn-resourcemanager - 2.3.4.0-3485 hadoop-yarn-timelineserver - 2.3.4.0-3485 hbase-client - 2.3.4.0-3485 hbase-master - 2.3.4.0-3485 hbase-regionserver - 2.3.4.0-3485 hive-metastore - None hive-server2 - None hive-webhcat - None kafka-broker - None knox-server - None mahout-client - None oozie-client - None oozie-server - None phoenix-client - 2.3.4.0-3485 phoenix-server - 2.3.4.0-3485 ranger-admin - None ranger-kms - None ranger-usersync - None slider-client - None spark-client - None spark-historyserver - None spark-thriftserver - None sqoop-client - None sqoop-server - None storm-client - None storm-nimbus - None storm-slider-client - None storm-supervisor - None zookeeper-client - 2.3.4.0-3485 zookeeper-server - 2.3.4.0-3485

And

hdp-select versions

2.3.2.0-2950

2.3.4.0-3485

Re: after upgrade hdp.

mjohnson — Mon, 18 Jul 2016 22:05:22 GMT

@Ashnee Sharma

When you gripped the ambari-agent.log for HBASE_CLIENT, did you see the error which Ambari is highlingting? If you could provide the output for the command 'grep HBASE_CLIENT /var/log/ambari-agent/amber_agent.log' for each of the services where HBASE_CLIENT is installed that would help to diagnosis the issue.

Re: after upgrade hdp.

ashneesharma88 — Mon, 18 Jul 2016 22:14:43 GMT

I got the error:-resource_management.core.exceptions.Fail: Execution of 'conf-select set-conf-dir --package hbase --stack-version 2.3.4.0-3485 --conf-version 0' returned 1. /usr/hdp/2.3.4.0-3485/hbase/conf does not exist

but if i checked on server files are there

ll /usr/hdp/2.3.4.0-3485/hbase/conf

total 44 -rw-r--r-- 1 hbase hadoop 2533 Jul 18 11:15 core-site.xml -rw-r--r-- 1 hbase root 2357 Jul 18 11:15 hadoop-metrics2-hbase.properties -rw-r--r-- 1 hbase root 2916 Jul 18 11:15 hbase-env.sh -rw-r--r-- 1 hbase hadoop 401 Jul 18 11:15 hbase-policy.xml -rw-r--r-- 1 hbase hadoop 5152 Jul 18 11:15 hbase-site.xml -rw-r--r-- 1 hbase hadoop 6568 Jul 18 11:15 hdfs-site.xml -rw-r--r-- 1 hbase hadoop 4235 Jul 18 11:15 log4j.properties -rw-r--r-- 1 hbase root 24 Jul 18 11:15 regionservers

Re: after upgrade hdp.

ashneesharma88 — Tue, 19 Jul 2016 15:03:41 GMT

output:-

grep HBASE_CLIENT /var/log/ambari-agent/ambari-agent.log

INFO 2016-07-18 09:51:18,734 ActionQueue.py:112 - Adding EXECUTION_COMMAND for role HBASE_CLIENT for service HBASE of cluster qacluster to the queue. INFO 2016-07-18 09:51:18,767 ActionQueue.py:232 - Executing command with id = 43-0 for role = HBASE_CLIENT of cluster qacluster. INFO 2016-07-18 09:58:00,631 ActionQueue.py:112 - Adding EXECUTION_COMMAND for role HBASE_CLIENT for service HBASE of cluster qacluster to the queue. INFO 2016-07-18 09:58:00,667 ActionQueue.py:232 - Executing command with id = 48-0 for role = HBASE_CLIENT of cluster qacluster. INFO 2016-07-18 11:15:52,649 ActionQueue.py:112 - Adding EXECUTION_COMMAND for role HBASE_CLIENT for service HBASE of cluster qacluster to the queue. INFO 2016-07-18 11:15:52,684 ActionQueue.py:232 - Executing command with id = 59-0 for role = HBASE_CLIENT of cluster qacluster.

And in ambari logs:-

On host qacluster.resilinc.com role HBASE_CLIENT in invalid state.
Invalid transition. Invalid event: HOST_SVCCOMP_OP_IN_PROGRESS at INSTALL_FAILED

Re: after upgrade hdp.

ashneesharma88 — Tue, 19 Jul 2016 17:57:47 GMT

I got the issue. After restarting hbase services from ambari hbase services are going down in 2 min. I checked in on server and i found its not creating PID file in /var/run/hbase. Any idea about this ?

Re: after upgrade hdp.

ashneesharma88 — Tue, 19 Jul 2016 22:20:48 GMT

I tried to reinstall the hbase service, then found the issue was with /hbase.distro. In this the HDP version was showing 2.4.2.0-258 and my hdp version is 2.3.4.0-3485. I did following stpes:-

vim /usr/hdp/current/hbase-client/bin/hbase +14

and replace the version :%s/2.4.2.0-258/2.3.4.0-3485/g

!wq

and tried to start the hbase and hbase started successfully.

🙂

Re: after upgrade hdp.

mjohnson — Tue, 19 Jul 2016 22:57:17 GMT

And that resolved the issue?

Re: after upgrade hdp.

ashneesharma88 — Fri, 22 Jul 2016 13:48:36 GMT

Yes issue is resolved..!!!!:)