Hive metatstore won't start after Ambari Blueprint deployment - using PosstgreSQL datastore

kc7bfi · ‎02-08-2018

I am deploying my cluster using an Anbari Blueprint. Everything works but the Hive Metatstore will not start. I get the following error:

2018-02-08 11:29:58,480 INFO  [main]: metastore.MetaStoreDirectSql (MetaStoreDirectSql.java:<init>(163)) - Using direct SQL, underlying DB is OTHER
2018-02-08 11:29:58,485 INFO  [main]: metastore.ObjectStore (ObjectStore.java:setConf(297)) - Initialized ObjectStore
2018-02-08 11:29:58,896 WARN  [main]: metastore.ObjectStore (ObjectStore.java:getDatabase(698)) - Failed to get database default, returning NoSuchObjectException
2018-02-08 11:29:59,351 ERROR [main]: hive.log (MetaStoreUtils.java:logAndThrowMetaException(1254)) - Got exception: java.io.IOException Couldn't create proxy provider class org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
java.io.IOException: Couldn't create proxy provider class org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
        at org.apache.hadoop.hdfs.NameNodeProxies.createFailoverProxyProvider(NameNodeProxies.java:526)
        at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:171)
        at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:690)
        at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:631)
        at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:160)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2795)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:99)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2829)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2811)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:390)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:179)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:374)
        at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
        at org.apache.hadoop.hive.metastore.Warehouse.getFs(Warehouse.java:106)
        at org.apache.hadoop.hive.metastore.Warehouse.getDnsPath(Warehouse.java:142)
        at org.apache.hadoop.hive.metastore.Warehouse.getDnsPath(Warehouse.java:148)
        at org.apache.hadoop.hive.metastore.Warehouse.getWhRoot(Warehouse.java:161)
        at org.apache.hadoop.hive.metastore.Warehouse.getDefaultDatabasePath(Warehouse.java:174)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB_core(HiveMetaStore.java:628)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:647)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:433)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105)
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:78)
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:91)
        at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:6396)
        at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:6391)
        at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:6658)
        at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:6575)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

Notice that it declares the underlying DB as "OTHER" not "POSTGRESQL". My Hive configuration is:

{
"hive-env": {
"properties": {
"hive_database": "Existing PostgreSQL Database",
"hive_database_name": "hive",
"hive_user": "hive",
"hive_database_type": "postgres",
"hive_ambari_database": "PostgreSQL",
"enable_hive_interactive": "false"
}
}
},
{
"hive-site": {
"properties": {
"javax.jdo.option.ConnectionDriverName": "org.postgresql.Driver",
"javax.jdo.option.ConnectionURL": "jdbc:postgresql://vicads-server.vicads5.local:5432/hive",
"javax.jdo.option.ConnectionUserName": "hive",
"javax.jdo.option.ConnectionPassword": "password",
"hive.zookeeper.quorum": "%HOSTGROUP::vicads_server_1%:2181,%HOSTGROUP::vicads_server_2%:2181,%HOSTGROUP::vicads_vault_1%:2181",
"hive.cluster.delegation.token.store.zookeeper.connectString": "%HOSTGROUP::vicads_server_1%:2181,%HOSTGROUP::vicads_server_2%:2181,%HOSTGROUP::vicads_vault_1%:2181",
"hive.metastore.uris": "thrift://%HOSTGROUP::vicads_server_1%:9083"
}
}
},

Am I doing something wrong? I do see that it creates the Hive tables in the PostgreSQL database. David

eric_deleforter · ‎10-23-2018

Hi David

Did you solve this problem ?

Best regards.

Eric.

lebrun_clement · ‎10-24-2018

Hello Eric,

We have met the same issue and the root cause was the absence of HDFS client on the metastore server.

Regards,

kc7bfi · ‎10-24-2018

Yes, but I don't remember the specifics.

eric_deleforter · ‎10-24-2018

Hello Clement and David

Yes, the Failed to get database default, returning NoSuchObjectException is not a very clear message, this mean that the default Hive database on hdfs is not reachable : hdfs://clustername/user/hive/warehouse/

So this is not a postgresql problem but a hdfs client problem.

Regards