Hi @bgooley ! thanks for replying back. sorry for the delay since this COVID-19 situation i can't get to the cluster for troubleshooting steps until now. To follow up the test using the keystore and truststore of each cluster CM, i found that JKS files in both clusters CM are contains 2 entries : PrivateKeyEntry and trustedCertEntry with Certificate fingerprint (SHA1) completely identical, only differs in alias name Target BDR JKS content: Keystore type: JKS Keystore provider: SUN Your keystore contains 2 entries servercert, Dec 20, 2019, PrivateKeyEntry, Certificate fingerprint (SHA1): D8:41:3A:3E:D8:9B:81:0E:D8:49:F7:3E:F1:93:52:C5:FC:BB:E7:CE cmdrc.company.co.id, Dec 20, 2019, trustedCertEntry, Certificate fingerprint (SHA1): D8:41:3A:3E:D8:9B:81:0E:D8:49:F7:3E:F1:93:52:C5:FC:BB:E7:CE   Source BDR JKS content: Keystore type: JKS Keystore provider: SUN Your keystore contains 2 entries servercert, Jan 16, 2020, PrivateKeyEntry, Certificate fingerprint (SHA1): D8:41:3A:3E:D8:9B:81:0E:D8:49:F7:3E:F1:93:52:C5:FC:BB:E7:CE cmprod.company.co.id, Jan 16, 2020, trustedCertEntry, Certificate fingerprint (SHA1): D8:41:3A:3E:D8:9B:81:0E:D8:49:F7:3E:F1:93:52:C5:FC:BB:E7:CE While on truststore on each cluster CM, entries with same Certificate fingerprint (SHA1) were found also : Target BDR truststore jssecacerts content: rootca, Jan 16, 2020, trustedCertEntry, Certificate fingerprint (SHA1): D8:41:3A:3E:D8:9B:81:0E:D8:49:F7:3E:F1:93:52:C5:FC:BB:E7:CE   Source BDR truststore jssecacerts content: rootca2, Dec 20, 2019, trustedCertEntry, Certificate fingerprint (SHA1): D8:41:3A:3E:D8:9B:81:0E:D8:49:F7:3E:F1:93:52:C5:FC:BB:E7:CE   For another background, the process of setting TLS/SSL on both cluster are a bit different from Cloudera docs. Since the cluster share the same domain and security team already had signed certificate from trusted CA, we  were provided with signed wildcard certificate in .PEM and . KEY files. so instead of generate JKS file from Java keytool on Cloudera nodes, we executes this steps : Create p12 file by combining signed certificate CA from root to local PEM files with corresponding KEY files (openssl pkcs12 -export command) Convert p12 file to JKS file with name of each hostname (keytool -importkeystore -destkeystore command) Import the root CA certificate into the JDK truststore jssecacerts (keytool -importcert command) Also another question, is BDR are sufficient with Level 1: Encryption only TLS/SSL setup on both cluster or is it require more (Level 2 :server-side certificate validation or even Level 3: dual certificate validation)?   Many thanks, Al ... View more

Running CDH 6.3.x. The workarounds with removed .scratchdir or referred in https://issues.cloudera.org/browse/HUE-8910 (added random UUID to directory name i.e. .scratchdir.<UUID>) do not solve the problem. Unfortunately do not have access to the knowledge base. Hence how to solve the problem? ... View more

@GangWar I did what you asked and i get this on the agent:   [11/May/2020 05:22:34 +0000] 6495 MainThread agent INFO To override these variables, use /etc/cloudera-scm-agent/config.ini. Environment variables for CDH locations are not used when CDH is installed from parcels. [11/May/2020 05:22:36 +0000] 6495 MainThread supervisor INFO Trying to connect to supervisor (Attempt 1) [11/May/2020 05:22:36 +0000] 6495 MainThread supervisor INFO Supervisor version: 3.0, pid: 1614 [11/May/2020 05:22:36 +0000] 6495 MainThread supervisor INFO Successfully connected to supervisor [11/May/2020 05:22:36 +0000] 6495 MainThread agent INFO Supervisor version: 3.0, pid: 1614 [11/May/2020 05:22:36 +0000] 6495 MainThread agent INFO Connecting to previous supervisor: agent-1614-1589173066. [11/May/2020 05:22:38 +0000] 6495 MainThread supervisor INFO Triggering supervisord update. [11/May/2020 05:22:38 +0000] 6495 MainThread _cplogging INFO [11/May/2020:05:22:38] ENGINE Bus STARTING [11/May/2020 05:22:38 +0000] 6495 MainThread _cplogging INFO [11/May/2020:05:22:38] ENGINE Started monitor thread '_TimeoutMonitor'. [11/May/2020 05:22:38 +0000] 6495 MainThread _cplogging INFO [11/May/2020:05:22:38] ENGINE Serving on http://127.0.0.1:9001 [11/May/2020 05:22:38 +0000] 6495 MainThread _cplogging INFO [11/May/2020:05:22:38] ENGINE Bus STARTED [11/May/2020 05:22:40 +0000] 6495 MainThread daemon INFO New monitor: (<cmf.monitor.host.HostMonitor object at 0x7f46e2d12ed0>,) [11/May/2020 05:22:40 +0000] 6495 MonitorDaemon-Scheduler daemon INFO Monitor ready to report: ('HostMonitor',) [11/May/2020 05:22:40 +0000] 6495 MainThread agent INFO Setting default socket timeout to 45 [11/May/2020 05:22:40 +0000] 6495 MainThread agent INFO Failed to read available parcel file: [Errno 2] No such file or directory: '/var/lib/cloudera-scm-agent/active_parcels.json' [11/May/2020 05:22:40 +0000] 6495 MainThread agent INFO Loading last saved hb response to complete initialization: /var/lib/cloudera-scm-agent/response.avro [11/May/2020 05:22:40 +0000] 6495 Monitor-HostMonitor network_interfaces INFO NIC iface ens5 doesn't support ETHTOOL (95) [11/May/2020 05:22:40 +0000] 6495 MainThread heartbeat_tracker INFO HB stats (seconds): num:1 LIFE_MIN:0.02 min:0.02 mean:0.02 max:0.02 LIFE_MAX:0.02 [11/May/2020 05:22:40 +0000] 6495 MainThread agent INFO CM server guid: 513d3669-b5a8-49c0-863a-c0396dff5c7b [11/May/2020 05:22:40 +0000] 6495 MainThread agent INFO Using parcels directory from server provided value: /opt/cloudera/parcels [11/May/2020 05:22:40 +0000] 6495 MainThread parcel INFO Agent does create users/groups and apply file permissions [11/May/2020 05:22:40 +0000] 6495 MainThread downloader INFO Downloader path: /opt/cloudera/parcel-cache [11/May/2020 05:22:40 +0000] 6495 MainThread parcel_cache INFO Using /opt/cloudera/parcel-cache for parcel cache [11/May/2020 05:22:40 +0000] 6495 MainThread throttling_logger WARNING Failed parsing alternatives line: rename string index out of range link best version is /usr/bin/file-rename [11/May/2020 05:22:40 +0000] 6495 MainThread agent INFO Flood daemon (re)start attempt [11/May/2020 05:22:42 +0000] 6495 MainThread firehoses INFO Reporting interval updated: 5.0 -> 60 [11/May/2020 05:22:42 +0000] 6495 MainThread agent ERROR Failed to handle Heartbeat Response: {u'firehoses': [{u'rol [big response....] ----------------------- Traceback (most recent call last): File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1528, in handle_heartbeat_response self._handle_heartbeat_response(response) File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1661, in _handle_heartbeat_response self._update_parcel_activation_state(response) File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1572, in _update_parcel_activation_state manage_old_parcels = old_response.get("create_parcel_symlinks") AttributeError: 'NoneType' object has no attribute 'get'   ... View more

Hi @abhagi,   Thanks for confirming this and great to know the issue got resolved! Please mark this thread as resolved when you get a chance.   Cheers, Li ... View more

Hi @lwang ,   Thank you very much for your reply, it answered all the questions I had.   Kind regards, Julius ... View more

Hi @TR7_BRYLE ,   Great to hear the issue is resolved! Thanks for sharing the solution.   Cheers, Li ... View more

Hi @lwang ,   Thanks it works .   Regards Bharad ... View more

Hi @manjj,   Thanks for reporting back the progress. There is a possible leaking somewhere in Hive even this bug was already fixed.   You can follow below steps if you do want to have the Hive Metastore canary test turned on. Steps: If Hive Metastore canary test is disabled, re-enable the Hive Metastore canary test. For configurations of both HDFS and Hive, find Service Monitor Client Config Overrides and add an entry for "fs.file.impl.disable.cache" with value "true". Restart Service Monitor And observe whether the heap also stays stable, with the canary back on. Thanks and hope this helps, Li ... View more