I think you need to verify the yarn and spark resources are configured properly. If yes then go and check from spark ui, it will show driver memory and executor memory. It is coming as expected then safely you can ignore it. ... View more

Hi @KPG1 , As per the description I believe you are looking for a way to have a high availability for HMS , if that is the case please find the below doc which can help -    https://docs.cloudera.com/cdp-private-cloud-upgrade/latest/upgrade-cdh/topics/hive-hms-ha-configuration.html   And for the backend DB to configure it for HA , it will be out of scope of Cloudera and you have to work with DB team itself to understand how they can configure.  ... View more

@KPG1 Here is a good post i found that describes the required steps to move postgres to mysql including a php to do the postgres to mysql conversion.   https://cloudera.ericlin.me/2016/10/hive-metastore-migration-from-embedded-postgres-to-mysql/     You may need to find a more modern tool for that conversion, but the process to dump, convert, and move metastores should be very similar.   ... View more

@KPG1 I can think of Oracle database ha a feature i.e Oracle RAC,where we give SCAN  address as the jdbc string to metastore,so the scan ip wll take care of loadbalancing.   You need to check with mysql vendor(Oracle),if they have similar thing as of Oracle database.   You can have HMS HA but all the HMS instance point to same database, you can not have 2 HMS instance pointing to a different database.   Please let me know,if you have any querties and please "Accept As Solution", if your queries are answered ... View more

@KPG1 We only support upgrading an existing cluster using Ambari or Cloudera Manager, instead of importing/updating the jars manually. In latest CDP Private cloud base, and our Public Cloud,  we are using Hadoop version 3.1.1 at this point. ... View more

Hello @KPG1 ,   The time taken to mark a datanode as stale is give by dfs.namenode.stale.datanode.interval, with a default of 30 seconds. If this is happening with a specific Datanode, you can check if there is any network issues between the Datanode and the Namenode or if the Datanode has any JVM pauses reported by checking the Datanode logs. As a bandaid, you can bump up the above parameter till the underlying problem is solved. ... View more