About smruti

smruti · ‎10-10-2021

@Shab Set client socket timeout to 300( 5 mins). The value for "hive.metastore.event.listeners" needs to be set to blank(no value).

smruti · ‎10-10-2021

@Shab no, those warn messages are not related. Did you try setting 'hive.metastore.client.socket.timeout' value to, 5 mins, under the "Service Monitor Client Config Overrides" in Hive configuration? We can completely disable Hive metastore canary, but I won't advise that. if Hive is working as expected, we could safely ignore these canary warnings, however, try setting the aforementioned socket timeout parameter, and that could help the canary tests perform without fail. If you use Sentry in your cluster, sometimes it blocks HMS, which in turn affects Metastore Canary. You could add the following property to overcome that. Navigate to Cloudera Manager > Hive > Configuration > "Hive Metastore Server Advanced Configuration Snippet (Safety Valve) for hive-site.xml" > add the following: <property> <name>hive.metastore.event.listeners</name> <description>This will stop HMS blocking by Sentry</description> <value></value> </property>

smruti · ‎10-07-2021

@hxn We should not have any hiveserver2 under "Hive" service. hiveserver2 should be present only under "Hive on Tez". If you have a hiveserver2 instance under "Hive" service, please delete it.

smruti · ‎10-07-2021

@dr If it's a managed table, you could get its size from TABLE_PARAMS table: e.g. SELECT a.TBL_NAME AS `TABLE`, b.PARAM_VALUE AS `SIZE` from TABLE_PARAMS b INNER JOIN TBLS a where a.TBL_ID=b.TBL_ID and b.PARAM_KEY='totalSize'; You could change the you need it. But, if there are external tables, or the table stats are not generated regularly, then you might not get the correct data. You could get the table size using HDFS file system commands as well: hdfs dfs -du -s -h <path to the table location> This will give you more accurate data.

smruti · ‎10-07-2021

@hxn It seems like hive on Tez service is not up. Please go through Hive on Tez logs, and you should see more info around the error 'Error stopping schq'. We need to bring Hive service up first, and then we can approach the Hue issue, or Hue might start working after fixing Hive.

smruti · ‎10-07-2021

@sat_046 Please refer to the Cloudera doc on Starting and stopping services.

smruti · ‎10-07-2021

@Shab are we talking about the "Metastore canary failed to create a database" error or the warning messages you have pasted here? If we are addressing the canary error, please check the Service Monitor logs if you find any relevant error or warning messages. If you notice any timeout error, you could increase 'hive.metastore.client.socket.timeout ' value to, say, 5 mins, under the "Service Monitor Client Config Overrides" in Hive configuration. Or if it's a different error, please paste them here. Service Monitor log file location : /var/log/cloudera-scm-firehose/ Refer to https://community.cloudera.com/t5/Support-Questions/The-Hive-Metastore-canary-failed-to-create-a-database/td-p/81021, if that helps.

smruti · ‎10-04-2021

@enirys You could refer to the following doc for Hive tuning: https://docs.cloudera.com/HDPDocuments/HDP2/HDP-2.6.5/bk_hive-performance-tuning/content/ch_connectivity-admission-control.html#guidelines-hiveserver2-heaps If you have other services running on the same HS2 node as well, you might want to reduce Hive heap size or move a service to a different node. Are you expecting too many connections as explained in the above doc, else you might want to bring down HS2 heap size? If you do not see too many connections, but notice high heap usage, you might want to take a heap dump as @asish mentioned, and see if there is memory leak. Load balancing across HS2 could take place based on how you are accessing Hive. You could use zooKeeper based connection string.

smruti · ‎09-29-2021

@enirys htop lists every single thread as a separate process. Every individual connection to HS2 and HMS shows up as different processes. So, you do not need to worry about that. As you say, you have 3 hiveserver2 nodes, see if you could reduce heap size of the hiveserver2. You could also lower 'hive.server2.thrift.max.worker.threads' so, it does not spin up too many threads in a single hs2 node. Make sure that your workload is getting distributed across HS2s.

smruti · ‎09-28-2021

Hi @DamienO As you say you are scanning Hive schemas, this must be putting a lot of load on Hive metastore. The driver logs suggest there are issues establishing new connections to Metastore. org.apache.hadoop.hive.ql.metadata.HiveException:java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient:33:1 You could review HS2 logs to see how many concurrent connections were there to the HMS at the time of the issue. Check if there is scope to increase max_connection value in your rdbms, or increase Metastore Heap size to accommodate more number of connections.

Online	Offline
Last Visited	‎02-13-2025 12:14 PM

Member Since	‎10-28-2020 05:19 AM
Last Visited	‎02-13-2025 12:14 PM
Posts	571
Kudos received	45

Cloudera Community

Re: ANALYZE command not write data into hive metas...

Re: HBase stores base64 data when data is inserted...

Re: Deleting hive service on CDP Private Base and ...

Re: Not Able to run import command. it fails with ...

Re: Any alternate for org.apache.hive:hive-jdbc ma...

Re: The Hive Metastore canary failed to create a d...

Re: The Hive Metastore canary failed to create a d...

Re: Unable to Connect Hive with Hue in CDH7.1

Re: Find table's size in Hive metastore (MySQL)

Re: Unable to Connect Hive with Hue in CDH7.1

Re: Services Restart Strategy for Cloudera Hadoop ...

Re: The Hive Metastore canary failed to create a d...

Re: Hive memory stuffing

Re: Hive memory stuffing

Re: Hortonworks ODBC Driver Crashes - STATUS_STACK...