Created on 08-03-2020 05:03 AM - edited 08-03-2020 07:24 AM
I need your help in figuring out why this Hive Metastore and Hive Server 2 are not working. Please find the ambari-error.log below:
Traceback (most recent call last):
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 995, in restart
self.status(env)
File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive_metastore.py", line 87, in status
check_process_status(status_params.hive_metastore_pid)
File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/check_process_status.py", line 43, in check_process_status
raise ComponentIsNotRunning()
ComponentIsNotRunning
The above exception was the cause of the following exception:
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive_metastore.py", line 201, in <module>
HiveMetastore().execute()
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute
method(env)
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 1006, in restart
self.start(env, upgrade_type=upgrade_type)
File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive_metastore.py", line 61, in start
create_metastore_schema() # execute without config lock
File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive.py", line 487, in create_metastore_schema
user = params.hive_user
File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
self.env.run()
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 263, in action_run
returns=self.resource.returns)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 72, in inner
result = function(command, **kwargs)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 102, in checked_call
tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy, returns=returns)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 150, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 314, in _call
raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of 'export HIVE_CONF_DIR=/usr/hdp/current/hive-metastore/conf/ ; /usr/hdp/current/hive-server2/bin/schematool -initSchema -dbType mysql -userName ambari_user -passWord [PROTECTED] -verbose' returned 1. SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/3.1.4.0-315/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/3.1.4.0-315/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Initializing the schema to: 3.1.1000
Metastore connection URL: jdbc:mysql://fgpoc.ambarisvr.com/
Metastore Connection Driver : com.mysql.jdbc.Driver
Metastore connection User: ambari_user
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
Starting metastore schema initialization to 3.1.1000
Initialization script hive-schema-3.1.1000.mysql.sql
Connecting to jdbc:mysql://fgpoc.ambarisvr.com/
Connected to: MySQL (version 5.7.31-0ubuntu0.16.04.1)
Driver: MySQL Connector/J (version mysql-connector-java-8.0.21 (Revision: 33f65445a1bcc544eb0120491926484da168f199))
Transaction isolation: TRANSACTION_READ_COMMITTED
0: jdbc:mysql://fgpoc.ambarisvr.com/> !autocommit on
Autocommit status: true
0: jdbc:mysql://fgpoc.ambarisvr.com/> /*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@C
HARACTER_SET_CLIENT */
No rows affected (0.01 seconds)
0: jdbc:mysql://fgpoc.ambarisvr.com/> /*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@
CHARACTER_SET_RESULTS */
No rows affected (0.001 seconds)
0: jdbc:mysql://fgpoc.ambarisvr.com/> /*!40101 SET @OLD_COLLATION_CONNECTION=@@C
OLLATION_CONNECTION */
No rows affected (0.001 seconds)
0: jdbc:mysql://fgpoc.ambarisvr.com/> /*!40101 SET NAMES utf8 */
No rows affected (0.002 seconds)
0: jdbc:mysql://fgpoc.ambarisvr.com/> /*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */
No rows affected (0.001 seconds)
0: jdbc:mysql://fgpoc.ambarisvr.com/> /*!40103 SET TIME_ZONE='+00:00' */
No rows affected (0.001 seconds)
0: jdbc:mysql://fgpoc.ambarisvr.com/> /*!40014 SET @OLD_UNIQUE_CHECKS=@@UNIQUE_C
HECKS, UNIQUE_CHECKS=0 */
No rows affected (0.002 seconds)
0: jdbc:mysql://fgpoc.ambarisvr.com/> /*!40014 SET @OLD_FOREIGN_KEY_CHECKS=@@FOR
EIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0 */
No rows affected (0.001 seconds)
0: jdbc:mysql://fgpoc.ambarisvr.com/> /*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL
_MODE='NO_AUTO_VALUE_ON_ZERO' */
No rows affected (0 seconds)
0: jdbc:mysql://fgpoc.ambarisvr.com/> /*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, S
QL_NOTES=0 */
No rows affected (0.001 seconds)
0: jdbc:mysql://fgpoc.ambarisvr.com/> /*!40101 SET @saved_cs_client = @@char
acter_set_client */
No rows affected (0.005 seconds)
0: jdbc:mysql://fgpoc.ambarisvr.com/> /*!40101 SET character_set_client = utf8 *
/
No rows affected (0.002 seconds)
0: jdbc:mysql://fgpoc.ambarisvr.com/> CREATE TABLE IF NOT EXISTS `BUCKETING_COLS
` ( `SD_ID` bigint(20) NOT NULL, `BUCKET_COL_NAME` varchar(256) CHARACTER SET la
tin1 COLLATE latin1_bin DEFAULT NULL, `INTEGER_IDX` int(11) NOT NULL, PRIMARY KE
Y (`SD_ID`,`INTEGER_IDX`), KEY `BUCKETING_COLS_N49` (`SD_ID`), CONSTRAINT `BUCKE
TING_COLS_FK1` FOREIGN KEY (`SD_ID`) REFERENCES `SDS` (`SD_ID`) ) ENGINE=InnoDB
DEFAULT CHARSET=latin1
Error: No database selected (state=3D000,code=1046)
Closing: 0: jdbc:mysql://fgpoc.ambarisvr.com/
Schema initialization FAILED! Metastore state would be inconsistent!
Underlying cause: java.io.IOException : Schema script failed, errorcode 2
org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization FAILED! Metastore state would be inconsistent!
at org.apache.hadoop.hive.metastore.tools.SchemaToolTaskInit.execute(SchemaToolTaskInit.java:66)
at org.apache.hadoop.hive.metastore.tools.MetastoreSchemaTool.run(MetastoreSchemaTool.java:446)
at org.apache.hive.beeline.schematool.HiveSchemaTool.main(HiveSchemaTool.java:138)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
Caused by: java.io.IOException: Schema script failed, errorcode 2
at org.apache.hive.beeline.schematool.HiveSchemaTool.execSql(HiveSchemaTool.java:105)
at org.apache.hive.beeline.schematool.HiveSchemaTool.execSql(HiveSchemaTool.java:83)
at org.apache.hadoop.hive.metastore.tools.SchemaToolTaskInit.execute(SchemaToolTaskInit.java:62)
... 8 more
*** schemaTool failed ***
Created on 08-08-2020 12:45 AM - edited 08-08-2020 12:48 AM
The problem was related to the database not properly configured.
We reset the ambari-server and configured the ambari-server again and that worked for us. To be sure if the database is configured properly, make sure in the hive database connection JDBC URL, the database name is appended, if not this might cause the issues.
Created 08-03-2020 08:46 PM
Created 08-04-2020 03:46 AM
I believe the syntax is not correct. Can you please run below command and try:
+++++
CREATE TABLE IF NOT EXISTS test_ext
( NAME string,
city string
)
partitioned by(ID int)
clustered by (NAME) into 3 buckets
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
STORED AS ORC
TBLPROPERTIES ("transactional"="true");
+++++
Is it HDP or CDH or CDP ? What is the version ?
Created on 08-06-2020 02:50 AM - edited 08-06-2020 02:52 AM
This script is not run manually. This script runs when we restart the Hive Metastore on Ambari. We are using HDP.
Can you help?
Created 08-06-2020 04:59 AM
You mean to say Hiveserver2 and Hivemetatsore is not starting?
If my understanding is correct we need hiveserver2 and hivemetastore log after reproducing the issue?
What changes made i.e causing the issue?
Created on 08-08-2020 12:45 AM - edited 08-08-2020 12:48 AM
The problem was related to the database not properly configured.
We reset the ambari-server and configured the ambari-server again and that worked for us. To be sure if the database is configured properly, make sure in the hive database connection JDBC URL, the database name is appended, if not this might cause the issues.