Support Questions

Find answers, ask questions, and share your expertise

Hive Meta Store and Hive Server 2 Failed to Start

avatar
Contributor

@Shelton@deepesh1,

 

I need your help in figuring out why this Hive Metastore and Hive Server 2 are not working. Please find the ambari-error.log below:

 

Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 995, in restart
    self.status(env)
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive_metastore.py", line 87, in status
    check_process_status(status_params.hive_metastore_pid)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/check_process_status.py", line 43, in check_process_status
    raise ComponentIsNotRunning()
ComponentIsNotRunning

The above exception was the cause of the following exception:

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive_metastore.py", line 201, in <module>
    HiveMetastore().execute()
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute
    method(env)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 1006, in restart
    self.start(env, upgrade_type=upgrade_type)
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive_metastore.py", line 61, in start
    create_metastore_schema() # execute without config lock
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive.py", line 487, in create_metastore_schema
    user = params.hive_user
  File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
    self.env.run()
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 263, in action_run
    returns=self.resource.returns)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 72, in inner
    result = function(command, **kwargs)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 102, in checked_call
    tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy, returns=returns)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 150, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 314, in _call
    raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of 'export HIVE_CONF_DIR=/usr/hdp/current/hive-metastore/conf/ ; /usr/hdp/current/hive-server2/bin/schematool -initSchema -dbType mysql -userName ambari_user -passWord [PROTECTED] -verbose' returned 1. SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/3.1.4.0-315/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/3.1.4.0-315/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Initializing the schema to: 3.1.1000
Metastore connection URL:	 jdbc:mysql://fgpoc.ambarisvr.com/
Metastore Connection Driver :	 com.mysql.jdbc.Driver
Metastore connection User:	 ambari_user
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
Starting metastore schema initialization to 3.1.1000
Initialization script hive-schema-3.1.1000.mysql.sql
Connecting to jdbc:mysql://fgpoc.ambarisvr.com/
Connected to: MySQL (version 5.7.31-0ubuntu0.16.04.1)
Driver: MySQL Connector/J (version mysql-connector-java-8.0.21 (Revision: 33f65445a1bcc544eb0120491926484da168f199))
Transaction isolation: TRANSACTION_READ_COMMITTED
0: jdbc:mysql://fgpoc.ambarisvr.com/> !autocommit on
Autocommit status: true
0: jdbc:mysql://fgpoc.ambarisvr.com/> /*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@C 
HARACTER_SET_CLIENT */
No rows affected (0.01 seconds)
0: jdbc:mysql://fgpoc.ambarisvr.com/> /*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@ 
CHARACTER_SET_RESULTS */
No rows affected (0.001 seconds)
0: jdbc:mysql://fgpoc.ambarisvr.com/> /*!40101 SET @OLD_COLLATION_CONNECTION=@@C 
OLLATION_CONNECTION */
No rows affected (0.001 seconds)
0: jdbc:mysql://fgpoc.ambarisvr.com/> /*!40101 SET NAMES utf8 */
No rows affected (0.002 seconds)
0: jdbc:mysql://fgpoc.ambarisvr.com/> /*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */ 
No rows affected (0.001 seconds)
0: jdbc:mysql://fgpoc.ambarisvr.com/> /*!40103 SET TIME_ZONE='+00:00' */
No rows affected (0.001 seconds)
0: jdbc:mysql://fgpoc.ambarisvr.com/> /*!40014 SET @OLD_UNIQUE_CHECKS=@@UNIQUE_C 
HECKS, UNIQUE_CHECKS=0 */
No rows affected (0.002 seconds)
0: jdbc:mysql://fgpoc.ambarisvr.com/> /*!40014 SET @OLD_FOREIGN_KEY_CHECKS=@@FOR 
EIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0 */
No rows affected (0.001 seconds)
0: jdbc:mysql://fgpoc.ambarisvr.com/> /*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL 
_MODE='NO_AUTO_VALUE_ON_ZERO' */
No rows affected (0 seconds)
0: jdbc:mysql://fgpoc.ambarisvr.com/> /*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, S 
QL_NOTES=0 */
No rows affected (0.001 seconds)
0: jdbc:mysql://fgpoc.ambarisvr.com/> /*!40101 SET @saved_cs_client     = @@char 
acter_set_client */
No rows affected (0.005 seconds)
0: jdbc:mysql://fgpoc.ambarisvr.com/> /*!40101 SET character_set_client = utf8 * 
/
No rows affected (0.002 seconds)
0: jdbc:mysql://fgpoc.ambarisvr.com/> CREATE TABLE IF NOT EXISTS `BUCKETING_COLS 
` ( `SD_ID` bigint(20) NOT NULL, `BUCKET_COL_NAME` varchar(256) CHARACTER SET la 
tin1 COLLATE latin1_bin DEFAULT NULL, `INTEGER_IDX` int(11) NOT NULL, PRIMARY KE 
Y (`SD_ID`,`INTEGER_IDX`), KEY `BUCKETING_COLS_N49` (`SD_ID`), CONSTRAINT `BUCKE 
TING_COLS_FK1` FOREIGN KEY (`SD_ID`) REFERENCES `SDS` (`SD_ID`) ) ENGINE=InnoDB  
DEFAULT CHARSET=latin1
Error: No database selected (state=3D000,code=1046)
Closing: 0: jdbc:mysql://fgpoc.ambarisvr.com/
Schema initialization FAILED! Metastore state would be inconsistent!
Underlying cause: java.io.IOException : Schema script failed, errorcode 2
org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization FAILED! Metastore state would be inconsistent!
	at org.apache.hadoop.hive.metastore.tools.SchemaToolTaskInit.execute(SchemaToolTaskInit.java:66)
	at org.apache.hadoop.hive.metastore.tools.MetastoreSchemaTool.run(MetastoreSchemaTool.java:446)
	at org.apache.hive.beeline.schematool.HiveSchemaTool.main(HiveSchemaTool.java:138)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
Caused by: java.io.IOException: Schema script failed, errorcode 2
	at org.apache.hive.beeline.schematool.HiveSchemaTool.execSql(HiveSchemaTool.java:105)
	at org.apache.hive.beeline.schematool.HiveSchemaTool.execSql(HiveSchemaTool.java:83)
	at org.apache.hadoop.hive.metastore.tools.SchemaToolTaskInit.execute(SchemaToolTaskInit.java:62)
	... 8 more
*** schemaTool failed ***

 

1 ACCEPTED SOLUTION

avatar
Contributor

The problem was related to the database not properly configured.

 

We reset the ambari-server and configured the ambari-server again and that worked for us. To be sure if the database is configured properly, make sure in the hive database connection JDBC URL, the database name is appended, if not this might cause the issues.

View solution in original post

5 REPLIES 5

avatar
Contributor

@jsensharma,

 

Can you help?

avatar
Guru

I believe the syntax is not correct. Can you please run below command and try:

 

+++++

CREATE TABLE IF NOT EXISTS test_ext
( NAME string,
city string
)
partitioned by(ID int)
clustered by (NAME) into 3 buckets
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
STORED AS ORC
TBLPROPERTIES ("transactional"="true");

+++++

 

Is it HDP or CDH or CDP ? What is the version ?

avatar
Contributor

@asish,

This script is not run manually. This script runs when we restart the Hive Metastore on Ambari. We are using HDP.

 

@Geoffrey Shelton Okot,

 

Can you help?

avatar
Guru

You mean to say Hiveserver2 and Hivemetatsore is not starting?

If my understanding is correct we need hiveserver2 and hivemetastore log after reproducing the issue?

 

What changes made i.e causing the issue?

avatar
Contributor

The problem was related to the database not properly configured.

 

We reset the ambari-server and configured the ambari-server again and that worked for us. To be sure if the database is configured properly, make sure in the hive database connection JDBC URL, the database name is appended, if not this might cause the issues.