Created 02-08-2018 05:48 PM
I am deploying my cluster using an Anbari Blueprint. Everything works but the Hive Metatstore will not start. I get the following error:
2018-02-08 11:29:58,480 INFO [main]: metastore.MetaStoreDirectSql (MetaStoreDirectSql.java:<init>(163)) - Using direct SQL, underlying DB is OTHER 2018-02-08 11:29:58,485 INFO [main]: metastore.ObjectStore (ObjectStore.java:setConf(297)) - Initialized ObjectStore 2018-02-08 11:29:58,896 WARN [main]: metastore.ObjectStore (ObjectStore.java:getDatabase(698)) - Failed to get database default, returning NoSuchObjectException 2018-02-08 11:29:59,351 ERROR [main]: hive.log (MetaStoreUtils.java:logAndThrowMetaException(1254)) - Got exception: java.io.IOException Couldn't create proxy provider class org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider java.io.IOException: Couldn't create proxy provider class org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider at org.apache.hadoop.hdfs.NameNodeProxies.createFailoverProxyProvider(NameNodeProxies.java:526) at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:171) at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:690) at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:631) at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:160) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2795) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:99) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2829) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2811) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:390) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:179) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:374) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295) at org.apache.hadoop.hive.metastore.Warehouse.getFs(Warehouse.java:106) at org.apache.hadoop.hive.metastore.Warehouse.getDnsPath(Warehouse.java:142) at org.apache.hadoop.hive.metastore.Warehouse.getDnsPath(Warehouse.java:148) at org.apache.hadoop.hive.metastore.Warehouse.getWhRoot(Warehouse.java:161) at org.apache.hadoop.hive.metastore.Warehouse.getDefaultDatabasePath(Warehouse.java:174) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB_core(HiveMetaStore.java:628) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:647) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:433) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:78) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:91) at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:6396) at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:6391) at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:6658) at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:6575) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:233) at org.apache.hadoop.util.RunJar.main(RunJar.java:148) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
Notice that it declares the underlying DB as "OTHER" not "POSTGRESQL". My Hive configuration is:
{ "hive-env": { "properties": { "hive_database": "Existing PostgreSQL Database", "hive_database_name": "hive", "hive_user": "hive", "hive_database_type": "postgres", "hive_ambari_database": "PostgreSQL", "enable_hive_interactive": "false" } } }, { "hive-site": { "properties": { "javax.jdo.option.ConnectionDriverName": "org.postgresql.Driver", "javax.jdo.option.ConnectionURL": "jdbc:postgresql://vicads-server.vicads5.local:5432/hive", "javax.jdo.option.ConnectionUserName": "hive", "javax.jdo.option.ConnectionPassword": "password", "hive.zookeeper.quorum": "%HOSTGROUP::vicads_server_1%:2181,%HOSTGROUP::vicads_server_2%:2181,%HOSTGROUP::vicads_vault_1%:2181", "hive.cluster.delegation.token.store.zookeeper.connectString": "%HOSTGROUP::vicads_server_1%:2181,%HOSTGROUP::vicads_server_2%:2181,%HOSTGROUP::vicads_vault_1%:2181", "hive.metastore.uris": "thrift://%HOSTGROUP::vicads_server_1%:9083" } } },
Am I doing something wrong? I do see that it creates the Hive tables in the PostgreSQL database. David
Created 10-23-2018 03:13 PM
Hi David
Did you solve this problem ?
Best regards.
Eric.
Created 10-24-2018 02:44 PM
Hello Eric,
We have met the same issue and the root cause was the absence of HDFS client on the metastore server.
Regards,
Created 10-24-2018 02:27 PM
Yes, but I don't remember the specifics.
Created 10-24-2018 02:57 PM
Hello Clement and David
Yes, the Failed to get database default, returning NoSuchObjectException is not a very clear message, this mean that the default Hive database on hdfs is not reachable : hdfs://clustername/user/hive/warehouse/
So this is not a postgresql problem but a hdfs client problem.
Regards