Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Sentry Initialization Fails

Solved Go to solution

Sentry Initialization Fails

Explorer

Environment

 

Cloudera Enterprise 5.5.1

HDFS, Hive, Impala, Sentry, Solr, Spark, YARN, Zookeeper

 

I just installed and setup Hive, Impala, and Sentry altogether. Sentry was able to create postgres tables, but the service does not start. I get the following output when running the command:

 

"sudo sentry --command schema-tool -conffile /etc/sentry/conf/sentry-file.xml --dbType postgres --initSchema"

 

Error: Syntax error: Encountered "START" at line 1, column 1. (state=42X01,code=30000)

Closing: 0: jdbc:derby:;databaseName=sentry_store_db;create=true

org.apache.sentry.SentryUserException: Schema initialization failed! Metastore state would be inconsistent!!

*** Sentry schemaTool failed ***

 

I am having trouble finding any resources on this (some stuff related to Hive showing this error but not Sentry). Any ideas on next steps?

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Sentry Initialization Fails

Explorer

Had to rebuild sentry database too to get that started. That then solved the previous Hive canary error.

 

Complicated yet interesting.

4 REPLIES 4

Re: Sentry Initialization Fails

Explorer

Updated with full input/output

 

sudo sentry --command schema-tool --conffile /etc/sentry/conf/sentry-site.xml --dbType postgres --initSchema
Sentry store connection URL:     jdbc:derby:;databaseName=sentry_store_db;create=true
Sentry store Connection Driver :         org.apache.derby.jdbc.EmbeddedDriver
Sentry store connection User:    Sentry
Starting sentry store schema initialization to 1.5.0
Initialization script sentry-postgres-1.5.0.sql
Connecting to jdbc:derby:;databaseName=sentry_store_db;create=true
Connected to: Apache Derby (version 10.10.2.0 - (1582446))
Driver: Apache Derby Embedded JDBC Driver (version 10.10.2.0 - (1582446))
Transaction isolation: TRANSACTION_REPEATABLE_READ
Autocommit status: true
Error: Syntax error: Encountered "START" at line 1, column 1. (state=42X01,code=30000)
Closing: 0: jdbc:derby:;databaseName=sentry_store_db;create=true
org.apache.sentry.SentryUserException: Schema initialization FAILED! Metastore state would be inconsistent !!
*** Sentry schemaTool failed ***

Re: Sentry Initialization Fails

Explorer

I see a problem in Hive now - so perhaps this is where the issue lies.

 

 

[main]: Query for candidates of org.apache.hadoop.hive.metastore.model.MVersionTable and subclasses resulted in no possible candidates
Required table missing : ""VERSION"" in Catalog "" Schema "". DataNucleus requires this table to perform its persistence operations. Either your MetaData is incorrect, or you need to enable "datanucleus.autoCreateTables"
org.datanucleus.store.rdbms.exceptions.MissingTableException: Required table missing : ""VERSION"" in Catalog "" Schema "". DataNucleus requires this table to perform its persistence operations. Either your MetaData is incorrect, or you need to enable "datanucleus.autoCreateTables"
    at org.datanucleus.store.rdbms.table.AbstractTable.exists(AbstractTable.java:485)
    at org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.performTablesValidation(RDBMSStoreManager.java:3380)
    at org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.addClassTablesAndValidate(RDBMSStoreManager.java:3190)
    at org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.run(RDBMSStoreManager.java:2841)
    at org.datanucleus.store.rdbms.AbstractSchemaTransaction.execute(AbstractSchemaTransaction.java:122)
    at org.datanucleus.store.rdbms.RDBMSStoreManager.addClasses(RDBMSStoreManager.java:1605)
    at org.datanucleus.store.AbstractStoreManager.addClass(AbstractStoreManager.java:954)
    at org.datanucleus.store.rdbms.RDBMSStoreManager.getDatastoreClass(RDBMSStoreManager.java:679)
    at org.datanucleus.store.rdbms.query.RDBMSQueryUtils.getStatementForCandidates(RDBMSQueryUtils.java:408)
    at org.datanucleus.store.rdbms.query.JDOQLQuery.compileQueryFull(JDOQLQuery.java:947)
    at org.datanucleus.store.rdbms.query.JDOQLQuery.compileInternal(JDOQLQuery.java:370)
    at org.datanucleus.store.query.Query.executeQuery(Query.java:1744)
    at org.datanucleus.store.query.Query.executeWithArray(Query.java:1672)
    at org.datanucleus.store.query.Query.execute(Query.java:1654)
    at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:221)
    at org.apache.hadoop.hive.metastore.ObjectStore.getMSchemaVersion(ObjectStore.java:6957)
    at org.apache.hadoop.hive.metastore.ObjectStore.getMetaStoreSchemaVersion(ObjectStore.java:6941)
    at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:6899)
    at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:6883)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:98)
    at com.sun.proxy.$Proxy10.verifySchema(Unknown Source)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:575)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:623)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:464)
    at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:78)
    at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:84)
    at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5775)
    at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5770)
    at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:6022)
    at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:5947)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

Mar 10, 8:51:19.864 AM     ERROR     org.apache.hadoop.hive.metastore.HiveMetaStore     

[main]: MetaException(message:Version information not found in metastore. )
    at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:6902)
    at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:6883)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:98)
    at com.sun.proxy.$Proxy10.verifySchema(Unknown Source)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:575)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:623)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:464)
    at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:78)
    at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:84)
    at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5775)
    at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5770)
    at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:6022)
    at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:5947)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

Mar 10, 8:51:19.864 AM     ERROR     org.apache.hadoop.hive.metastore.HiveMetaStore     

[main]: Metastore Thrift Server threw an exception...
MetaException(message:Version information not found in metastore. )
    at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:6902)
    at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:6883)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:98)
    at com.sun.proxy.$Proxy10.verifySchema(Unknown Source)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:575)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:623)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:464)
    at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:78)
    at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:84)
    at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5775)
    at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5770)
    at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:6022)
    at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:5947)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

Mar 10, 8:55:20.286 AM     INFO     org.apache.hadoop.hive.metastore.HiveMetaStore     

[Thread-3]: Shutting down hive metastore.

 

 

 

Connecting to postgres I cannot find the metastore DB. But when I shut down Hive ,and then use the Cloudera Manager option to generate the metastore, it succeeds with "already exists".

Highlighted

Re: Sentry Initialization Fails

Explorer

It seems I was able to at least partially fix metastore DB in postgres.

 

Now it starts up a lot nicer according to logs. But now I get:

 

"The Hive Metastore canary failed to drop the table it created"

Re: Sentry Initialization Fails

Explorer

Had to rebuild sentry database too to get that started. That then solved the previous Hive canary error.

 

Complicated yet interesting.

Don't have an account?
Coming from Hortonworks? Activate your account here