Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hive Metastore won't start

Highlighted

Hive Metastore won't start

Contributor

Evaluating hortonwrokws distribution for the first time. Some services needed manual start but were ok at the end. Hive Metastore consistently fails on:

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py", line 211, in <module>
    HiveMetastore().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 329, in execute
    method(env)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 865, in restart
    self.start(env, upgrade_type=upgrade_type)
  File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py", line 61, in start
    create_metastore_schema()
  File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py", line 382, in create_metastore_schema
    user = params.hive_user
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 166, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 262, in action_run
    tries=self.resource.tries, try_sleep=self.resource.try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 72, in inner
    result = function(command, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 102, in checked_call
    tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 150, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 303, in _call
    raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of 'export HIVE_CONF_DIR=/usr/hdp/current/hive-metastore/conf/conf.server ; /usr/hdp/current/hive-server2-hive2/bin/schematool -initSchema -dbType mysql -userName hive -passWord [PROTECTED] -verbose' returned 1. SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/2.6.2.0-205/hive2/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.6.2.0-205/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL:	 jdbc:mysql://il-dev-05/hive?createDatabaseIfNotExist=true
Metastore Connection Driver :	 com.mysql.jdbc.Driver
Metastore connection User:	 hive
org.apache.hadoop.hive.metastore.HiveMetaException: Failed to get schema version.
Underlying cause: java.sql.SQLException : null,  message from server: "Host 'il-dev-05' is not allowed to connect to this MySQL server"
SQL Error code: 1130
org.apache.hadoop.hive.metastore.HiveMetaException: Failed to get schema version.
	at org.apache.hive.beeline.HiveSchemaHelper.getConnectionToMetastore(HiveSchemaHelper.java:80)
	at org.apache.hive.beeline.HiveSchemaTool.getConnectionToMetastore(HiveSchemaTool.java:133)
	at org.apache.hive.beeline.HiveSchemaTool.testConnectionToMetastore(HiveSchemaTool.java:187)
	at org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:291)
	at org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:277)
	at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:526)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
Caused by: java.sql.SQLException: null,  message from server: "Host 'il-dev-05' is not allowed to connect to this MySQL server"
	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:959)
	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:898)
	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:887)
	at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1038)
	at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2254)
	at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2285)
	at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2084)
	at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:795)
	at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:44)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)
	at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:400)
	at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:327)
	at java.sql.DriverManager.getConnection(DriverManager.java:664)
	at java.sql.DriverManager.getConnection(DriverManager.java:247)
	at org.apache.hive.beeline.HiveSchemaHelper.getConnectionToMetastore(HiveSchemaHelper.java:76)
	... 11 more
*** schemaTool failed ***
6 REPLIES 6
Highlighted

Re: Hive Metastore won't start

Expert Contributor

@Yair Ogen

As per the log "message from server:"Host 'il-dev-05' is not allowed to connect to this MySQL server", user "hive"@"metastorehost" does not have a valid userid on mysql server at il-dev-05. Create a user on mysql with "hive"@"metastorehost" and provide full access to hive database.

You can refer here - https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.2.0/bk_ambari-administration/content/using_hive...

Highlighted

Re: Hive Metastore won't start

Mentor

@Yair Ogen

This is typically a permission issue. At the time you create the hive/mysql user you should have included the below in your grant commands.

You will need to log into mysql as root and run the below code where " il-dev-05". is the FQDN of your metastorehost

GRANT ALL PRIVILEGES ON *.* TO 'hive'@'localhost'; 
GRANT ALL PRIVILEGES ON *.* TO 'hive'@'%'; 
GRANT ALL PRIVILEGES ON *.* TO 'hive'@'il-dev-05'; 
GRANT ALL PRIVILEGES ON hive.* TO 'hive'@'localhost' IDENTIFIED BY 'hive' WITH GRANT OPTION; 
GRANT ALL PRIVILEGES ON hive.* TO 'hive'@'il-dev-05' IDENTIFIED BY 'hive' WITH GRANT OPTION; 
FLUSH PRIVILEGES; 
quit;

After executing successfully the above your metastore should start without any issues

Highlighted

Re: Hive Metastore won't start

Contributor

I found this link:

https://discuss.pivotal.io/hc/en-us/articles/218434858-HiveServer2-fails-to-start-because-host-is-no...

This resolved my issue after I run this followed by pwd reset:

rm -rf /var/lib/mysql/*

mysqld --initialize --explicit_defaults_for_timestamp

sudo mysqld_safe --skip-grant-tables &

update user set authentication_string=password('system') where user='root';

FLUSH PRIVILEGES;

quit
Highlighted

Re: Hive Metastore won't start

Mentor

@Yair Ogen

Uum I still fail to understand how the root password change resolved a hive user issue?No corelation

Re: Hive Metastore won't start

Contributor

@Geoffrey Shelton Okot

the root password change was just something I had to do as the mysql was not letting me in with the default credentials
The add user for hive/change privileges, as they suggest in the above mentioned link, really solved the issue

Highlighted

Re: Hive Metastore won't start

Mentor

@Yair Ogen

That re-assures me because had you run the same GRANTS statements I had sent it would have also resolved. I was keen to know why you had to change the root password!

Enjoy

Don't have an account?
Coming from Hortonworks? Activate your account here