CDH 5.1 Hive Metastore Server can't stay up

[#1]

CDH 5.1 is installed using Cloudera Manager 5.1. Oracle database is used for Hive Metastore and tables were populated after install. However, Hive Metastore Server was not started. I followed instruction in the installation guide to onfigure the Hive Metastore and edit hive-site.xml. After a restart, Hive Metastore Server stays up for about 5 minutes and then goes down. During that 5 minutes, There are about 11 hive user sessions connecting to the database. Below is the messages I found in the log:

2014-09-05 10:23:15,587 INFO org.apache.hadoop.hive.metastore.HiveMetaStore: Starting hive metastore on port 9083
2014-09-05 10:23:15,768 WARN org.apache.hadoop.hive.conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore.
2014-09-05 10:23:15,864 INFO org.apache.hadoop.hive.metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
2014-09-05 10:23:15,901 INFO org.apache.hadoop.hive.metastore.ObjectStore: ObjectStore, initialize called
2014-09-05 10:23:16,222 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored
2014-09-05 10:23:16,644 WARN com.jolbox.bonecp.BoneCPConfig: Max Connections < 1. Setting to 20
2014-09-05 10:23:17,453 WARN org.apache.hadoop.hive.conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore.
2014-09-05 10:23:17,453 INFO org.apache.hadoop.hive.metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
2014-09-05 10:23:17,502 WARN com.jolbox.bonecp.BoneCPConfig: Max Connections < 1. Setting to 20
2014-09-05 10:23:17,611 INFO org.apache.hadoop.hive.metastore.MetaStoreDirectSql: MySQL check failed, assuming we are not on mysql: ORA-00922: missing or invalid option

2014-09-05 10:23:21,465 INFO org.apache.hadoop.hive.metastore.ObjectStore: Initialized ObjectStore

[#2]

Thanks for your help. I increased java heap size of hive metastore server in bytes to 6GB. It just took longer (25 minutes) for the process to die. The resident memory usage was 5.8GB.

I noticed that the runtime hive-site.xml has hive.metastore.local set to true and there is no hive.metastore.uris property. We are using Oracle as the data store. Is this normal?

[#3]

By default, CM configures /etc/hive/conf/hive-site.xml to talk to the metastore. Setting hive.metastore.local to true is generally a bad idea, and means your clients will talk to the database directly instead of going through the Hive Metastore Server (HMS). This can be controlled by the flag "Bypass Hive Metastore Server" in Hive configuration.

It's quite strange that your HMS would get so much memory when your clients are not talking to it. You might want to try debugging this memory usage on the Hive forums.

There's nothing wrong with using an Oracle database.

[#4]

Bypass Hive Metastore Server is unchecked. I wonder why the runtime hive-site.xml has hive.metastore.local set to true. The server has 32GB RAM, but I don't think allocating more java heap size would help. It seems to have memory leak.

[#5]

You may just need to deploy client configuration. Until you do so, /etc/hive/conf can be stale.

You'll need to ask on hive forums to understand the memory issue.

[#6]

I deployed Hive client configuration files and /etc/hive/conf/hive-site.xml timestamp is up to date. The file has defined hive.metastore.uris. However, the runtime hive-site.xml does not hve hive.metastore.uris. The stderr of HMS shows "Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore". Any ideas how to resolve this issue?

[#7]

What do you mean by "the runtime hive-site.xml"? How are you running hive? Normally if you run "hive" from the command-line, it'll pick up /etc/hive/conf/hive-site.xml (on the same host as where you ran "hive").

[#8]

I restart HMS via Cloudera Manager. On the page of HMS/processes, the hive-site.xml listed under Configuration Files is what I meant runtime hive-site.xm, which has different contents than /etc/hive/conf/hive-site.xml.

[#9]

I see what you mean now. The HMS is supposed to have different configuration. This is the one process that is supposed to talk to the relational database, so it sets "hive.metastore.local" to true and doesn't need to set the URIs (the URIs would just point to itself!). This is expected and not the cause of your memory issue.

[#10 · Accepted solution]

The problem has been resolved by deleting Hive service and adding it back. The health test of HMS failed initially - the metastore canary failed to drop the table it created. Then it became good after a few minutes. The "show tables" command works in the Hive shell now.

The only mistake or cause I can think of is that we started panic after seeing the red/bad health test result right after the install. Then, we troubleshot and edited hive-site.xml by following Configuring the Hive Metastore instruction in the CDH installation guide, which is not necessary and probably caused the issue.

[#11]

Thank you for your help. I was able to increase java heap size of hive metatstore server in bytes to 6GB. It just took longer (25 minutes) for the process to die. The usage of resident memory was up to 5.8GB.
I notice that the runtime hive-site.xml has hive.metastore.local set to true and hive.metastore.uris is not there. We use Oracle Oracle as metastore. Is this normal?