Skip to main content
Shab
Shab
Silver ⭐️⭐️
October 5, 2021
Solved

The Hive Metastore canary failed to create a database

Hello all,

I know this has been asked a lot of times before but I have not been able to resolve this issue using those fixes. I get this error twice or thrice every day. Someone please help.

 

 

3:44:45.647 PM WARN Configuration
[pool-8-thread-192]: org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@464eba80:an attempt to override final parameter: hive.exec.dynamic.partition.mode; Ignoring.
3:44:45.647 PM WARN Configuration
[pool-8-thread-192]: org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@464eba80:an attempt to override final parameter: hive.compactor.initiator.on; Ignoring.
3:44:45.648 PM WARN HiveConf
[pool-8-thread-192]: HiveConf of name hive.sentry.conf.url does not exist
3:44:45.648 PM WARN HiveConf
[pool-8-thread-192]: HiveConf of name hive.enforce.bucketing does not exist

 

Thanks,

Shaban

smruti
smrutiAnswer
Gold ⭐️⭐️⭐️
October 10, 2021

@Shab no, those warn messages are not related. Did you try setting 'hive.metastore.client.socket.timeout' value to, 5 mins, under the "Service Monitor Client Config Overrides" in Hive configuration?

 

We can completely disable Hive metastore canary, but I won't advise that. if Hive is working as expected, we could safely ignore these canary warnings, however, try setting the  aforementioned socket timeout parameter, and that could help the canary tests perform without fail.

 

If you use Sentry in your cluster, sometimes it blocks HMS, which in turn affects Metastore Canary. You could add the following property to overcome that.

 

Navigate to Cloudera Manager > Hive > Configuration > "Hive Metastore Server Advanced Configuration Snippet (Safety Valve) for hive-site.xml" > add the following: 
        <property> 
        <name>hive.metastore.event.listeners</name> 
        <description>This will stop HMS blocking by Sentry</description> 
        <value></value> 
        </property>

 

View in thread

6 replies

smruti
Gold ⭐️⭐️⭐️
smruti
Gold ⭐️⭐️⭐️
October 7, 2021

@Shab are we talking about the "Metastore canary failed to create a database" error or the warning messages you have pasted here? 

 

If we are addressing the canary error, please check the Service Monitor logs if you find any relevant error or warning messages.  If you notice any timeout error, you could increase 'hive.metastore.client.socket.timeout ' value to, say, 5 mins, under the "Service Monitor Client Config Overrides" in Hive configuration.

 

Or if it's a different error, please paste them here.

Service Monitor log file location : /var/log/cloudera-scm-firehose/

 

Refer to https://community.cloudera.com/t5/Support-Questions/The-Hive-Metastore-canary-failed-to-create-a-database/td-p/81021, if that helps.

Translate
Shab
ShabAuthor
Silver ⭐️⭐️
October 10, 2021

Thanks@smruti for your reply. I am getting these error alerts ("HIVEMETASTORE_CANARY_HEALTH has become bad: The Hive Metastore canary failed to create a database.") but when I look at the logs I see only these warnings that I had pasted in my initial question. I get these alerts 4 to 5 times a day, although it recovers in about 5 minutes every time. I checked the service monitor logs but I did not find any timeout errors there as well. Could these warnings be setting off alerts? If yes then can we stop that from happening?

 

Translate
smruti
Gold ⭐️⭐️⭐️
smrutiAnswer
Gold ⭐️⭐️⭐️
October 10, 2021

@Shab no, those warn messages are not related. Did you try setting 'hive.metastore.client.socket.timeout' value to, 5 mins, under the "Service Monitor Client Config Overrides" in Hive configuration?

 

We can completely disable Hive metastore canary, but I won't advise that. if Hive is working as expected, we could safely ignore these canary warnings, however, try setting the  aforementioned socket timeout parameter, and that could help the canary tests perform without fail.

 

If you use Sentry in your cluster, sometimes it blocks HMS, which in turn affects Metastore Canary. You could add the following property to overcome that.

 

Navigate to Cloudera Manager > Hive > Configuration > "Hive Metastore Server Advanced Configuration Snippet (Safety Valve) for hive-site.xml" > add the following: 
        <property> 
        <name>hive.metastore.event.listeners</name> 
        <description>This will stop HMS blocking by Sentry</description> 
        <value></value> 
        </property>

 

Translate
    Terms & ConditionsAccessibility statement