Created on 08-03-2020 05:03 AM - edited 08-03-2020 07:24 AM
I need your help in figuring out why this Hive Metastore and Hive Server 2 are not working. Please find the ambari-error.log below:
Traceback (most recent call last): File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 995, in restart self.status(env) File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive_metastore.py", line 87, in status check_process_status(status_params.hive_metastore_pid) File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/check_process_status.py", line 43, in check_process_status raise ComponentIsNotRunning() ComponentIsNotRunning The above exception was the cause of the following exception: Traceback (most recent call last): File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive_metastore.py", line 201, in <module> HiveMetastore().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute method(env) File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 1006, in restart self.start(env, upgrade_type=upgrade_type) File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive_metastore.py", line 61, in start create_metastore_schema() # execute without config lock File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive.py", line 487, in create_metastore_schema user = params.hive_user File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__ self.env.run() File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run self.run_action(resource, action) File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action provider_action() File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 263, in action_run returns=self.resource.returns) File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 72, in inner result = function(command, **kwargs) File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 102, in checked_call tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy, returns=returns) File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 150, in _call_wrapper result = _call(command, **kwargs_copy) File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 314, in _call raise ExecutionFailed(err_msg, code, out, err) resource_management.core.exceptions.ExecutionFailed: Execution of 'export HIVE_CONF_DIR=/usr/hdp/current/hive-metastore/conf/ ; /usr/hdp/current/hive-server2/bin/schematool -initSchema -dbType mysql -userName ambari_user -passWord [PROTECTED] -verbose' returned 1. SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/hdp/3.1.4.0-315/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/3.1.4.0-315/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] Initializing the schema to: 3.1.1000 Metastore connection URL: jdbc:mysql://fgpoc.ambarisvr.com/ Metastore Connection Driver : com.mysql.jdbc.Driver Metastore connection User: ambari_user Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary. Starting metastore schema initialization to 3.1.1000 Initialization script hive-schema-3.1.1000.mysql.sql Connecting to jdbc:mysql://fgpoc.ambarisvr.com/ Connected to: MySQL (version 5.7.31-0ubuntu0.16.04.1) Driver: MySQL Connector/J (version mysql-connector-java-8.0.21 (Revision: 33f65445a1bcc544eb0120491926484da168f199)) Transaction isolation: TRANSACTION_READ_COMMITTED 0: jdbc:mysql://fgpoc.ambarisvr.com/> !autocommit on Autocommit status: true 0: jdbc:mysql://fgpoc.ambarisvr.com/> /*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@C HARACTER_SET_CLIENT */ No rows affected (0.01 seconds) 0: jdbc:mysql://fgpoc.ambarisvr.com/> /*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@ CHARACTER_SET_RESULTS */ No rows affected (0.001 seconds) 0: jdbc:mysql://fgpoc.ambarisvr.com/> /*!40101 SET @OLD_COLLATION_CONNECTION=@@C OLLATION_CONNECTION */ No rows affected (0.001 seconds) 0: jdbc:mysql://fgpoc.ambarisvr.com/> /*!40101 SET NAMES utf8 */ No rows affected (0.002 seconds) 0: jdbc:mysql://fgpoc.ambarisvr.com/> /*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */ No rows affected (0.001 seconds) 0: jdbc:mysql://fgpoc.ambarisvr.com/> /*!40103 SET TIME_ZONE='+00:00' */ No rows affected (0.001 seconds) 0: jdbc:mysql://fgpoc.ambarisvr.com/> /*!40014 SET @OLD_UNIQUE_CHECKS=@@UNIQUE_C HECKS, UNIQUE_CHECKS=0 */ No rows affected (0.002 seconds) 0: jdbc:mysql://fgpoc.ambarisvr.com/> /*!40014 SET @OLD_FOREIGN_KEY_CHECKS=@@FOR EIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0 */ No rows affected (0.001 seconds) 0: jdbc:mysql://fgpoc.ambarisvr.com/> /*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL _MODE='NO_AUTO_VALUE_ON_ZERO' */ No rows affected (0 seconds) 0: jdbc:mysql://fgpoc.ambarisvr.com/> /*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, S QL_NOTES=0 */ No rows affected (0.001 seconds) 0: jdbc:mysql://fgpoc.ambarisvr.com/> /*!40101 SET @saved_cs_client = @@char acter_set_client */ No rows affected (0.005 seconds) 0: jdbc:mysql://fgpoc.ambarisvr.com/> /*!40101 SET character_set_client = utf8 * / No rows affected (0.002 seconds) 0: jdbc:mysql://fgpoc.ambarisvr.com/> CREATE TABLE IF NOT EXISTS `BUCKETING_COLS ` ( `SD_ID` bigint(20) NOT NULL, `BUCKET_COL_NAME` varchar(256) CHARACTER SET la tin1 COLLATE latin1_bin DEFAULT NULL, `INTEGER_IDX` int(11) NOT NULL, PRIMARY KE Y (`SD_ID`,`INTEGER_IDX`), KEY `BUCKETING_COLS_N49` (`SD_ID`), CONSTRAINT `BUCKE TING_COLS_FK1` FOREIGN KEY (`SD_ID`) REFERENCES `SDS` (`SD_ID`) ) ENGINE=InnoDB DEFAULT CHARSET=latin1 Error: No database selected (state=3D000,code=1046) Closing: 0: jdbc:mysql://fgpoc.ambarisvr.com/ Schema initialization FAILED! Metastore state would be inconsistent! Underlying cause: java.io.IOException : Schema script failed, errorcode 2 org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization FAILED! Metastore state would be inconsistent! at org.apache.hadoop.hive.metastore.tools.SchemaToolTaskInit.execute(SchemaToolTaskInit.java:66) at org.apache.hadoop.hive.metastore.tools.MetastoreSchemaTool.run(MetastoreSchemaTool.java:446) at org.apache.hive.beeline.schematool.HiveSchemaTool.main(HiveSchemaTool.java:138) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:318) at org.apache.hadoop.util.RunJar.main(RunJar.java:232) Caused by: java.io.IOException: Schema script failed, errorcode 2 at org.apache.hive.beeline.schematool.HiveSchemaTool.execSql(HiveSchemaTool.java:105) at org.apache.hive.beeline.schematool.HiveSchemaTool.execSql(HiveSchemaTool.java:83) at org.apache.hadoop.hive.metastore.tools.SchemaToolTaskInit.execute(SchemaToolTaskInit.java:62) ... 8 more *** schemaTool failed ***
Created on 08-08-2020 12:45 AM - edited 08-08-2020 12:48 AM
The problem was related to the database not properly configured.
We reset the ambari-server and configured the ambari-server again and that worked for us. To be sure if the database is configured properly, make sure in the hive database connection JDBC URL, the database name is appended, if not this might cause the issues.
Created 08-03-2020 08:46 PM
Created 08-04-2020 03:46 AM
I believe the syntax is not correct. Can you please run below command and try:
+++++
CREATE TABLE IF NOT EXISTS test_ext
( NAME string,
city string
)
partitioned by(ID int)
clustered by (NAME) into 3 buckets
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
STORED AS ORC
TBLPROPERTIES ("transactional"="true");
+++++
Is it HDP or CDH or CDP ? What is the version ?
Created on 08-06-2020 02:50 AM - edited 08-06-2020 02:52 AM
This script is not run manually. This script runs when we restart the Hive Metastore on Ambari. We are using HDP.
Can you help?
Created 08-06-2020 04:59 AM
You mean to say Hiveserver2 and Hivemetatsore is not starting?
If my understanding is correct we need hiveserver2 and hivemetastore log after reproducing the issue?
What changes made i.e causing the issue?
Created on 08-08-2020 12:45 AM - edited 08-08-2020 12:48 AM
The problem was related to the database not properly configured.
We reset the ambari-server and configured the ambari-server again and that worked for us. To be sure if the database is configured properly, make sure in the hive database connection JDBC URL, the database name is appended, if not this might cause the issues.