The Hive Metastore canary failed to create a database

[#1]

Hello all,

I know this has been asked a lot of times before but I have not been able to resolve this issue using those fixes. I get this error twice or thrice every day. Someone please help.

3:44:45.647 PM WARN Configuration
[pool-8-thread-192]: org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@464eba80:an attempt to override final parameter: hive.exec.dynamic.partition.mode; Ignoring.
3:44:45.647 PM WARN Configuration
[pool-8-thread-192]: org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@464eba80:an attempt to override final parameter: hive.compactor.initiator.on; Ignoring.
3:44:45.648 PM WARN HiveConf
[pool-8-thread-192]: HiveConf of name hive.sentry.conf.url does not exist
3:44:45.648 PM WARN HiveConf
[pool-8-thread-192]: HiveConf of name hive.enforce.bucketing does not exist

Thanks,

Shaban

[#2 · Accepted solution]

@Shab no, those warn messages are not related. Did you try setting 'hive.metastore.client.socket.timeout' value to, 5 mins, under the "Service Monitor Client Config Overrides" in Hive configuration?

We can completely disable Hive metastore canary, but I won't advise that. if Hive is working as expected, we could safely ignore these canary warnings, however, try setting the  aforementioned socket timeout parameter, and that could help the canary tests perform without fail.

If you use Sentry in your cluster, sometimes it blocks HMS, which in turn affects Metastore Canary. You could add the following property to overcome that.

Navigate to Cloudera Manager > Hive > Configuration > "Hive Metastore Server Advanced Configuration Snippet (Safety Valve) for hive-site.xml" > add the following: 
        <property> 
        <name>hive.metastore.event.listeners</name> 
        <description>This will stop HMS blocking by Sentry</description> 
        <value></value> 
        </property>

[#3]

@Shab are we talking about the "Metastore canary failed to create a database" error or the warning messages you have pasted here? 

If we are addressing the canary error, please check the Service Monitor logs if you find any relevant error or warning messages.  If you notice any timeout error, you could increase 'hive.metastore.client.socket.timeout ' value to, say, 5 mins, under the "Service Monitor Client Config Overrides" in Hive configuration.

Or if it's a different error, please paste them here.

Service Monitor log file location : /var/log/cloudera-scm-firehose/

Refer to https://community.cloudera.com/t5/Support-Questions/The-Hive-Metastore-canary-failed-to-create-a-dat... if that helps.

[#4]

Thanks@smruti for your reply. I am getting these error alerts ("HIVEMETASTORE_CANARY_HEALTH has become bad: The Hive Metastore canary failed to create a database.") but when I look at the logs I see only these warnings that I had pasted in my initial question. I get these alerts 4 to 5 times a day, although it recovers in about 5 minutes every time. I checked the service monitor logs but I did not find any timeout errors there as well. Could these warnings be setting off alerts? If yes then can we stop that from happening?

[#5 · Accepted solution]

@Shab no, those warn messages are not related. Did you try setting 'hive.metastore.client.socket.timeout' value to, 5 mins, under the "Service Monitor Client Config Overrides" in Hive configuration?

We can completely disable Hive metastore canary, but I won't advise that. if Hive is working as expected, we could safely ignore these canary warnings, however, try setting the  aforementioned socket timeout parameter, and that could help the canary tests perform without fail.

If you use Sentry in your cluster, sometimes it blocks HMS, which in turn affects Metastore Canary. You could add the following property to overcome that.

Navigate to Cloudera Manager > Hive > Configuration > "Hive Metastore Server Advanced Configuration Snippet (Safety Valve) for hive-site.xml" > add the following: 
        <property> 
        <name>hive.metastore.event.listeners</name> 
        <description>This will stop HMS blocking by Sentry</description> 
        <value></value> 
        </property>

[#6]

@smruti  'hive.metastore.client.socket.timeout' value is currently set to 60. Should I change it to 5? Or should I go ahead with the 'hive.metastore.event.listeners' setting you had suggested, as we do use Sentry in our cluster. What do you suggest should I set the value to in this case?

[#7 · Accepted solution]

@Shab Set client socket timeout to 300( 5 mins). 

The value for "hive.metastore.event.listeners" needs to be set to blank(no value).

[#8]

Thanks @smruti . Will try this and update with results.