2020-10-05 15:11:33,801 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.5.0-152, Upgrade Direction=upgrade -> 3.1.5.0-152
2020-10-05 15:11:33,817 - Using hadoop conf dir: /usr/hdp/3.1.5.0-152/hadoop/conf
2020-10-05 15:11:33,962 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.5.0-152, Upgrade Direction=upgrade -> 3.1.5.0-152
2020-10-05 15:11:33,966 - Using hadoop conf dir: /usr/hdp/3.1.5.0-152/hadoop/conf
2020-10-05 15:11:33,967 - Group['kms'] {}
2020-10-05 15:11:33,968 - Group['ranger'] {}
2020-10-05 15:11:33,968 - Group['hdfs'] {}
2020-10-05 15:11:33,969 - Group['hadoop'] {}
2020-10-05 15:11:33,969 - Group['users'] {}
2020-10-05 15:11:33,969 - Group['knox'] {}
2020-10-05 15:11:33,969 - User['yarn-ats'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2020-10-05 15:11:33,970 - User['hive'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2020-10-05 15:11:33,971 - User['infra-solr'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2020-10-05 15:11:33,972 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2020-10-05 15:11:33,973 - User['oozie'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'users'], 'uid': None}
2020-10-05 15:11:33,973 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2020-10-05 15:11:33,974 - User['ranger'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['ranger', 'hadoop'], 'uid': None}
2020-10-05 15:11:33,975 - User['tez'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'users'], 'uid': None}
2020-10-05 15:11:33,976 - User['kms'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['kms', 'hadoop'], 'uid': None}
2020-10-05 15:11:33,976 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'users'], 'uid': None}
2020-10-05 15:11:33,977 - User['hdfs'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hdfs', 'hadoop'], 'uid': None}
2020-10-05 15:11:33,978 - User['sqoop'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2020-10-05 15:11:33,979 - User['yarn'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2020-10-05 15:11:33,979 - User['mapred'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2020-10-05 15:11:33,980 - User['hbase'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2020-10-05 15:11:33,981 - User['knox'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'knox'], 'uid': None}
2020-10-05 15:11:33,982 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2020-10-05 15:11:33,983 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2020-10-05 15:11:33,989 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] due to not_if
2020-10-05 15:11:33,990 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'create_parents': True, 'mode': 0775, 'cd_access': 'a'}
2020-10-05 15:11:33,991 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2020-10-05 15:11:33,992 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2020-10-05 15:11:33,993 - call['/var/lib/ambari-agent/tmp/changeUid.sh hbase'] {}
2020-10-05 15:11:34,002 - call returned (0, '1024')
2020-10-05 15:11:34,002 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase 1024'] {'not_if': '(test $(id -u hbase) -gt 1000) || (false)'}
2020-10-05 15:11:34,008 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase 1024'] due to not_if
2020-10-05 15:11:34,009 - Group['hdfs'] {}
2020-10-05 15:11:34,009 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': ['hdfs', 'hadoop', u'hdfs']}
2020-10-05 15:11:34,010 - FS Type: HDFS
2020-10-05 15:11:34,010 - Directory['/etc/hadoop'] {'mode': 0755}
2020-10-05 15:11:34,020 - File['/usr/hdp/3.1.5.0-152/hadoop/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2020-10-05 15:11:34,021 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 01777}
2020-10-05 15:11:34,034 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'}
2020-10-05 15:11:34,042 - Skipping Execute[('setenforce', '0')] due to not_if
2020-10-05 15:11:34,042 - Directory['/var/log/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'hadoop', 'mode': 0775, 'cd_access': 'a'}
2020-10-05 15:11:34,044 - Directory['/var/run/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'root', 'cd_access': 'a'}
2020-10-05 15:11:34,045 - Changing owner for /var/run/hadoop from 1021 to root
2020-10-05 15:11:34,045 - Changing group for /var/run/hadoop from 1003 to root
2020-10-05 15:11:34,045 - Directory['/var/run/hadoop/hdfs'] {'owner': 'hdfs', 'cd_access': 'a'}
2020-10-05 15:11:34,046 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'create_parents': True, 'cd_access': 'a'}
2020-10-05 15:11:34,049 - File['/usr/hdp/3.1.5.0-152/hadoop/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'hdfs'}
2020-10-05 15:11:34,051 - File['/usr/hdp/3.1.5.0-152/hadoop/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'hdfs'}
2020-10-05 15:11:34,056 - File['/usr/hdp/3.1.5.0-152/hadoop/conf/log4j.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
2020-10-05 15:11:34,065 - File['/usr/hdp/3.1.5.0-152/hadoop/conf/hadoop-metrics2.properties'] {'content': Template('hadoop-metrics2.properties.j2'), 'owner': 'hdfs', 'group': 'hadoop'}
2020-10-05 15:11:34,065 - File['/usr/hdp/3.1.5.0-152/hadoop/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755}
2020-10-05 15:11:34,066 - File['/usr/hdp/3.1.5.0-152/hadoop/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'}
2020-10-05 15:11:34,069 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop', 'mode': 0644}
2020-10-05 15:11:34,074 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755}
2020-10-05 15:11:34,079 - Skipping unlimited key JCE policy check and setup since it is not required
2020-10-05 15:11:34,328 - Using hadoop conf dir: /usr/hdp/3.1.5.0-152/hadoop/conf
2020-10-05 15:11:34,328 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.5.0-152, Upgrade Direction=upgrade -> 3.1.5.0-152
2020-10-05 15:11:34,345 - Using hadoop conf dir: /usr/hdp/3.1.5.0-152/hadoop/conf
2020-10-05 15:11:34,358 - Directory['/etc/security/limits.d'] {'owner': 'root', 'create_parents': True, 'group': 'root'}
2020-10-05 15:11:34,361 - File['/etc/security/limits.d/hdfs.conf'] {'content': Template('hdfs.conf.j2'), 'owner': 'root', 'group': 'root', 'mode': 0644}
2020-10-05 15:11:34,362 - XmlConfig['hadoop-policy.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/3.1.5.0-152/hadoop/conf', 'mode': 0644, 'configuration_attributes': {}, 'owner': 'hdfs', 'configurations': ...}
2020-10-05 15:11:34,369 - Generating config: /usr/hdp/3.1.5.0-152/hadoop/conf/hadoop-policy.xml
2020-10-05 15:11:34,369 - File['/usr/hdp/3.1.5.0-152/hadoop/conf/hadoop-policy.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
2020-10-05 15:11:34,375 - XmlConfig['ssl-client.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/3.1.5.0-152/hadoop/conf', 'mode': 0644, 'configuration_attributes': {}, 'owner': 'hdfs', 'configurations': ...}
2020-10-05 15:11:34,381 - Generating config: /usr/hdp/3.1.5.0-152/hadoop/conf/ssl-client.xml
2020-10-05 15:11:34,381 - File['/usr/hdp/3.1.5.0-152/hadoop/conf/ssl-client.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
2020-10-05 15:11:34,385 - Directory['/usr/hdp/3.1.5.0-152/hadoop/conf/secure'] {'owner': 'root', 'create_parents': True, 'group': 'hadoop', 'cd_access': 'a'}
2020-10-05 15:11:34,386 - XmlConfig['ssl-client.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/3.1.5.0-152/hadoop/conf/secure', 'mode': 0644, 'configuration_attributes': {}, 'owner': 'hdfs', 'configurations': ...}
2020-10-05 15:11:34,393 - Generating config: /usr/hdp/3.1.5.0-152/hadoop/conf/secure/ssl-client.xml
2020-10-05 15:11:34,393 - File['/usr/hdp/3.1.5.0-152/hadoop/conf/secure/ssl-client.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
2020-10-05 15:11:34,397 - XmlConfig['ssl-server.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/3.1.5.0-152/hadoop/conf', 'mode': 0644, 'configuration_attributes': {}, 'owner': 'hdfs', 'configurations': ...}
2020-10-05 15:11:34,403 - Generating config: /usr/hdp/3.1.5.0-152/hadoop/conf/ssl-server.xml
2020-10-05 15:11:34,404 - File['/usr/hdp/3.1.5.0-152/hadoop/conf/ssl-server.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
2020-10-05 15:11:34,408 - XmlConfig['hdfs-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/3.1.5.0-152/hadoop/conf', 'mode': 0644, 'configuration_attributes': {}, 'owner': 'hdfs', 'configurations': ...}
2020-10-05 15:11:34,414 - Generating config: /usr/hdp/3.1.5.0-152/hadoop/conf/hdfs-site.xml
2020-10-05 15:11:34,414 - File['/usr/hdp/3.1.5.0-152/hadoop/conf/hdfs-site.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
2020-10-05 15:11:34,452 - XmlConfig['core-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/3.1.5.0-152/hadoop/conf', 'xml_include_file': None, 'mode': 0644, 'configuration_attributes': {}, 'owner': 'hdfs', 'configurations': ...}
2020-10-05 15:11:34,458 - Generating config: /usr/hdp/3.1.5.0-152/hadoop/conf/core-site.xml
2020-10-05 15:11:34,458 - File['/usr/hdp/3.1.5.0-152/hadoop/conf/core-site.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
2020-10-05 15:11:34,484 - File['/usr/hdp/3.1.5.0-152/hadoop/conf/slaves'] {'content': Template('slaves.j2'), 'owner': 'hdfs'}
2020-10-05 15:11:34,487 - Directory['/master/hadoop/hdfs/namenode'] {'owner': 'hdfs', 'group': 'hadoop', 'create_parents': True, 'mode': 0755, 'cd_access': 'a'}
2020-10-05 15:11:34,495 - Directory['/usr/lib/ambari-logsearch-logfeeder/conf'] {'create_parents': True, 'mode': 0755, 'cd_access': 'a'}
2020-10-05 15:11:34,496 - Generate Log Feeder config file: /usr/lib/ambari-logsearch-logfeeder/conf/input.config-hdfs.json
2020-10-05 15:11:34,496 - File['/usr/lib/ambari-logsearch-logfeeder/conf/input.config-hdfs.json'] {'content': Template('input.config-hdfs.json.j2'), 'mode': 0644}
2020-10-05 15:11:34,497 - Skipping setting up secure ZNode ACL for HFDS as it's supported only for secure clusters.
2020-10-05 15:11:34,500 - Called service start with upgrade_type: None
2020-10-05 15:11:34,500 - Ranger Hdfs plugin is not enabled
2020-10-05 15:11:34,502 - File['/etc/hadoop/conf/dfs.exclude'] {'owner': 'hdfs', 'content': Template('exclude_hosts_list.j2'), 'group': 'hadoop'}
2020-10-05 15:11:34,503 - Options for start command are: -upgrade
2020-10-05 15:11:34,503 - Directory['/var/run/hadoop'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 0755}
2020-10-05 15:11:34,503 - Changing owner for /var/run/hadoop from 0 to hdfs
2020-10-05 15:11:34,503 - Changing group for /var/run/hadoop from 0 to hadoop
2020-10-05 15:11:34,503 - Directory['/var/run/hadoop/hdfs'] {'owner': 'hdfs', 'group': 'hadoop', 'create_parents': True}
2020-10-05 15:11:34,504 - Directory['/var/log/hadoop/hdfs'] {'owner': 'hdfs', 'group': 'hadoop', 'create_parents': True}
2020-10-05 15:11:34,504 - File['/var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid'] {'action': ['delete'], 'not_if': 'ambari-sudo.sh  -H -E test -f /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid && ambari-sudo.sh  -H -E pgrep -F /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid'}
2020-10-05 15:11:34,519 - Deleting File['/var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid']
2020-10-05 15:11:34,519 - Execute['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'ulimit -c unlimited ;  /usr/hdp/3.1.5.0-152/hadoop/bin/hdfs --config /usr/hdp/3.1.5.0-152/hadoop/conf --daemon start namenode -upgrade''] {'environment': {'HADOOP_LIBEXEC_DIR': '/usr/hdp/3.1.5.0-152/hadoop/libexec'}, 'not_if': 'ambari-sudo.sh  -H -E test -f /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid && ambari-sudo.sh  -H -E pgrep -F /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid'}
2020-10-05 15:11:36,666 - Waiting for the NameNode to broadcast whether it is Active or Standby...
2020-10-05 15:11:36,668 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpEl1d9a 2>/tmp/tmpEUHefS''] {'quiet': False}
2020-10-05 15:11:36,734 - call returned (7, '')
2020-10-05 15:11:36,735 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:11:38 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:11:38,543 - call returned (255, '20/10/05 15:11:38 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:11:38,543 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpWPmyNr 2>/tmp/tmpYnGkLY''] {'quiet': False}
2020-10-05 15:11:38,605 - call returned (7, '')
2020-10-05 15:11:38,606 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:11:40 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:11:40,425 - call returned (255, '20/10/05 15:11:40 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:11:40,425 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:11:40,425 - Will retry 4 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:11:45,431 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpR9WZCe 2>/tmp/tmpndTFQJ''] {'quiet': False}
2020-10-05 15:11:45,491 - call returned (7, '')
2020-10-05 15:11:45,492 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:11:47 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:11:47,287 - call returned (255, '20/10/05 15:11:47 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:11:47,287 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpcAudWJ 2>/tmp/tmpen1NSk''] {'quiet': False}
2020-10-05 15:11:47,347 - call returned (7, '')
2020-10-05 15:11:47,348 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:11:49 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:11:49,140 - call returned (255, '20/10/05 15:11:49 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:11:49,140 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:11:49,140 - Will retry 3 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:11:54,146 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpF5ojkv 2>/tmp/tmp1fBxiX''] {'quiet': False}
2020-10-05 15:11:54,208 - call returned (7, '')
2020-10-05 15:11:54,209 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:11:56 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:11:56,032 - call returned (255, '20/10/05 15:11:56 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:11:56,033 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmppiNeyi 2>/tmp/tmpEFt4kw''] {'quiet': False}
2020-10-05 15:11:56,097 - call returned (7, '')
2020-10-05 15:11:56,098 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:11:57 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:11:57,939 - call returned (255, '20/10/05 15:11:57 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:11:57,939 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:11:57,939 - Will retry 2 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:12:02,945 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpgCDFJI 2>/tmp/tmpI8eN2K''] {'quiet': False}
2020-10-05 15:12:03,004 - call returned (7, '')
2020-10-05 15:12:03,005 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:12:04 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:12:04,810 - call returned (255, '20/10/05 15:12:04 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:12:04,810 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpmbtBvG 2>/tmp/tmprR6QS5''] {'quiet': False}
2020-10-05 15:12:04,870 - call returned (7, '')
2020-10-05 15:12:04,871 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:12:06 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:12:06,696 - call returned (255, '20/10/05 15:12:06 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:12:06,696 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:12:06,696 - Will retry 1 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:12:11,702 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpcPQthS 2>/tmp/tmpZ8_wjs''] {'quiet': False}
2020-10-05 15:12:11,762 - call returned (7, '')
2020-10-05 15:12:11,763 - Getting jmx metrics from NN failed. URL: http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmpcPQthS 2>/tmp/tmpZ8_wjs' returned 7. 

2020-10-05 15:12:11,763 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:12:13 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:12:13,564 - call returned (255, '20/10/05 15:12:13 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:12:13,565 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpdmhWsP 2>/tmp/tmpSFl_ox''] {'quiet': False}
2020-10-05 15:12:13,624 - call returned (7, '')
2020-10-05 15:12:13,625 - Getting jmx metrics from NN failed. URL: http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmpdmhWsP 2>/tmp/tmpSFl_ox' returned 7. 

2020-10-05 15:12:13,625 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:12:15 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:12:15,419 - call returned (255, '20/10/05 15:12:15 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:12:15,420 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:12:15,420 - No active NameNode was found after 5 retries. Will return current NameNode HA states
2020-10-05 15:12:15,420 - Will retry 124 time(s), caught exception: The NameNode nn1 is not listed as Active or Standby, waiting.... Sleeping for 5 sec(s)
2020-10-05 15:12:20,426 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpb6lz8x 2>/tmp/tmpH7T8hU''] {'quiet': False}
2020-10-05 15:12:20,487 - call returned (7, '')
2020-10-05 15:12:20,488 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:12:22 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:12:22,332 - call returned (255, '20/10/05 15:12:22 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:12:22,333 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpw8ieuS 2>/tmp/tmp08k4zz''] {'quiet': False}
2020-10-05 15:12:22,395 - call returned (7, '')
2020-10-05 15:12:22,395 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:12:24 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:12:24,249 - call returned (255, '20/10/05 15:12:24 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:12:24,249 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:12:24,249 - Will retry 4 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:12:29,255 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpNmQZsq 2>/tmp/tmpiuFQat''] {'quiet': False}
2020-10-05 15:12:29,316 - call returned (7, '')
2020-10-05 15:12:29,317 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:12:31 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:12:31,146 - call returned (255, '20/10/05 15:12:31 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:12:31,146 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmphp0iLt 2>/tmp/tmpLvz1at''] {'quiet': False}
2020-10-05 15:12:31,206 - call returned (7, '')
2020-10-05 15:12:31,206 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:12:32 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:12:33,005 - call returned (255, '20/10/05 15:12:32 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:12:33,005 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:12:33,005 - Will retry 3 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:12:38,011 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmprkUWV1 2>/tmp/tmp1Yalnx''] {'quiet': False}
2020-10-05 15:12:38,071 - call returned (7, '')
2020-10-05 15:12:38,071 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:12:39 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:12:39,874 - call returned (255, '20/10/05 15:12:39 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:12:39,875 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp3M1DP0 2>/tmp/tmpiJOWxN''] {'quiet': False}
2020-10-05 15:12:39,936 - call returned (7, '')
2020-10-05 15:12:39,937 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:12:41 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:12:41,733 - call returned (255, '20/10/05 15:12:41 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:12:41,733 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:12:41,734 - Will retry 2 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:12:46,739 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpvaArZq 2>/tmp/tmpoLYakS''] {'quiet': False}
2020-10-05 15:12:46,801 - call returned (7, '')
2020-10-05 15:12:46,802 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:12:48 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:12:48,622 - call returned (255, '20/10/05 15:12:48 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:12:48,622 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp4UDHUD 2>/tmp/tmpt8WxnD''] {'quiet': False}
2020-10-05 15:12:48,688 - call returned (7, '')
2020-10-05 15:12:48,688 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:12:50 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:12:50,486 - call returned (255, '20/10/05 15:12:50 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:12:50,487 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:12:50,487 - Will retry 1 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:12:55,490 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpFuvVzM 2>/tmp/tmp9aL9QP''] {'quiet': False}
2020-10-05 15:12:55,550 - call returned (7, '')
2020-10-05 15:12:55,551 - Getting jmx metrics from NN failed. URL: http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmpFuvVzM 2>/tmp/tmp9aL9QP' returned 7. 

2020-10-05 15:12:55,551 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:12:57 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:12:57,343 - call returned (255, '20/10/05 15:12:57 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:12:57,344 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpgUPQaB 2>/tmp/tmpHBhfU2''] {'quiet': False}
2020-10-05 15:12:57,404 - call returned (7, '')
2020-10-05 15:12:57,405 - Getting jmx metrics from NN failed. URL: http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmpgUPQaB 2>/tmp/tmpHBhfU2' returned 7. 

2020-10-05 15:12:57,405 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:12:59 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:12:59,218 - call returned (255, '20/10/05 15:12:59 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:12:59,218 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:12:59,218 - No active NameNode was found after 5 retries. Will return current NameNode HA states
2020-10-05 15:12:59,219 - Will retry 123 time(s), caught exception: The NameNode nn1 is not listed as Active or Standby, waiting.... Sleeping for 5 sec(s)
2020-10-05 15:13:04,224 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpjUeTry 2>/tmp/tmpdWeuh9''] {'quiet': False}
2020-10-05 15:13:04,286 - call returned (7, '')
2020-10-05 15:13:04,286 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:13:06 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:13:06,084 - call returned (255, '20/10/05 15:13:06 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:13:06,084 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp_dhzMN 2>/tmp/tmp8d3LOx''] {'quiet': False}
2020-10-05 15:13:06,145 - call returned (7, '')
2020-10-05 15:13:06,145 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:13:07 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:13:07,974 - call returned (255, '20/10/05 15:13:07 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:13:07,974 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:13:07,974 - Will retry 4 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:13:12,980 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp76ETzm 2>/tmp/tmphXnmn6''] {'quiet': False}
2020-10-05 15:13:13,043 - call returned (7, '')
2020-10-05 15:13:13,044 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:13:14 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:13:14,891 - call returned (255, '20/10/05 15:13:14 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:13:14,891 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpGa3RkU 2>/tmp/tmpp1QnTk''] {'quiet': False}
2020-10-05 15:13:14,953 - call returned (7, '')
2020-10-05 15:13:14,954 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:13:16 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:13:16,749 - call returned (255, '20/10/05 15:13:16 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:13:16,749 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:13:16,749 - Will retry 3 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:13:21,755 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpwD3q0y 2>/tmp/tmp5Xq0I5''] {'quiet': False}
2020-10-05 15:13:21,815 - call returned (7, '')
2020-10-05 15:13:21,816 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:13:23 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:13:23,613 - call returned (255, '20/10/05 15:13:23 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:13:23,613 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpRJ7Qx8 2>/tmp/tmpBkGSQk''] {'quiet': False}
2020-10-05 15:13:23,674 - call returned (7, '')
2020-10-05 15:13:23,674 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:13:25 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:13:25,521 - call returned (255, '20/10/05 15:13:25 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:13:25,521 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:13:25,521 - Will retry 2 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:13:30,524 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpNlola8 2>/tmp/tmp4DqZEG''] {'quiet': False}
2020-10-05 15:13:30,584 - call returned (7, '')
2020-10-05 15:13:30,585 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:13:32 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:13:32,381 - call returned (255, '20/10/05 15:13:32 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:13:32,382 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpxofuSE 2>/tmp/tmp3qvN1j''] {'quiet': False}
2020-10-05 15:13:32,443 - call returned (7, '')
2020-10-05 15:13:32,444 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:13:34 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:13:34,251 - call returned (255, '20/10/05 15:13:34 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:13:34,251 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:13:34,251 - Will retry 1 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:13:39,257 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpKtYzko 2>/tmp/tmpho75JH''] {'quiet': False}
2020-10-05 15:13:39,324 - call returned (7, '')
2020-10-05 15:13:39,324 - Getting jmx metrics from NN failed. URL: http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmpKtYzko 2>/tmp/tmpho75JH' returned 7. 

2020-10-05 15:13:39,325 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:13:41 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:13:41,254 - call returned (255, '20/10/05 15:13:41 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:13:41,255 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpgJu4Ak 2>/tmp/tmpaPM2ti''] {'quiet': False}
2020-10-05 15:13:41,321 - call returned (7, '')
2020-10-05 15:13:41,321 - Getting jmx metrics from NN failed. URL: http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmpgJu4Ak 2>/tmp/tmpaPM2ti' returned 7. 

2020-10-05 15:13:41,321 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:13:43 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:13:43,171 - call returned (255, '20/10/05 15:13:43 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:13:43,171 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:13:43,171 - No active NameNode was found after 5 retries. Will return current NameNode HA states
2020-10-05 15:13:43,172 - Will retry 122 time(s), caught exception: The NameNode nn1 is not listed as Active or Standby, waiting.... Sleeping for 5 sec(s)
2020-10-05 15:13:48,177 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmprWHQ7B 2>/tmp/tmp41CU3Q''] {'quiet': False}
2020-10-05 15:13:48,238 - call returned (7, '')
2020-10-05 15:13:48,238 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:13:50 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:13:50,041 - call returned (255, '20/10/05 15:13:50 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:13:50,041 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmptlUYTr 2>/tmp/tmptUmBaS''] {'quiet': False}
2020-10-05 15:13:50,101 - call returned (7, '')
2020-10-05 15:13:50,101 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:13:51 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:13:51,901 - call returned (255, '20/10/05 15:13:51 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:13:51,901 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:13:51,901 - Will retry 4 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:13:56,907 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpogIr1E 2>/tmp/tmpIh5tAI''] {'quiet': False}
2020-10-05 15:13:56,968 - call returned (7, '')
2020-10-05 15:13:56,968 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:13:58 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:13:58,768 - call returned (255, '20/10/05 15:13:58 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:13:58,769 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp1P7TtT 2>/tmp/tmpgjogwx''] {'quiet': False}
2020-10-05 15:13:58,831 - call returned (7, '')
2020-10-05 15:13:58,831 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:14:00 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:14:00,667 - call returned (255, '20/10/05 15:14:00 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:14:00,668 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:14:00,668 - Will retry 3 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:14:05,673 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpbjciB2 2>/tmp/tmp9DyueJ''] {'quiet': False}
2020-10-05 15:14:05,733 - call returned (7, '')
2020-10-05 15:14:05,734 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:14:07 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:14:07,528 - call returned (255, '20/10/05 15:14:07 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:14:07,528 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpXFbVIP 2>/tmp/tmp6dhxO8''] {'quiet': False}
2020-10-05 15:14:07,588 - call returned (7, '')
2020-10-05 15:14:07,588 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:14:09 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:14:09,387 - call returned (255, '20/10/05 15:14:09 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:14:09,387 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:14:09,387 - Will retry 2 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:14:14,393 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmprrvrwo 2>/tmp/tmpBZB6FN''] {'quiet': False}
2020-10-05 15:14:14,453 - call returned (7, '')
2020-10-05 15:14:14,453 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:14:16 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:14:16,247 - call returned (255, '20/10/05 15:14:16 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:14:16,248 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp9sGjO4 2>/tmp/tmpLGwtaN''] {'quiet': False}
2020-10-05 15:14:16,308 - call returned (7, '')
2020-10-05 15:14:16,308 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:14:18 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:14:18,120 - call returned (255, '20/10/05 15:14:18 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:14:18,120 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:14:18,120 - Will retry 1 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:14:23,125 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpLRZq6H 2>/tmp/tmpEahUSv''] {'quiet': False}
2020-10-05 15:14:23,185 - call returned (7, '')
2020-10-05 15:14:23,186 - Getting jmx metrics from NN failed. URL: http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmpLRZq6H 2>/tmp/tmpEahUSv' returned 7. 

2020-10-05 15:14:23,186 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:14:25 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:14:25,035 - call returned (255, '20/10/05 15:14:25 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:14:25,036 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpXcbYIN 2>/tmp/tmpFyqh1E''] {'quiet': False}
2020-10-05 15:14:25,099 - call returned (7, '')
2020-10-05 15:14:25,100 - Getting jmx metrics from NN failed. URL: http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmpXcbYIN 2>/tmp/tmpFyqh1E' returned 7. 

2020-10-05 15:14:25,100 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:14:26 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:14:26,931 - call returned (255, '20/10/05 15:14:26 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:14:26,931 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:14:26,931 - No active NameNode was found after 5 retries. Will return current NameNode HA states
2020-10-05 15:14:26,932 - Will retry 121 time(s), caught exception: The NameNode nn1 is not listed as Active or Standby, waiting.... Sleeping for 5 sec(s)
2020-10-05 15:14:31,938 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpfs_RCG 2>/tmp/tmptc7mHV''] {'quiet': False}
2020-10-05 15:14:31,997 - call returned (7, '')
2020-10-05 15:14:31,997 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:14:33 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:14:33,801 - call returned (255, '20/10/05 15:14:33 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:14:33,802 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpNrDUZn 2>/tmp/tmpJewqUU''] {'quiet': False}
2020-10-05 15:14:33,864 - call returned (7, '')
2020-10-05 15:14:33,865 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:14:35 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:14:35,692 - call returned (255, '20/10/05 15:14:35 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:14:35,692 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:14:35,693 - Will retry 4 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:14:40,699 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpuAZFbw 2>/tmp/tmpXVwtc6''] {'quiet': False}
2020-10-05 15:14:40,758 - call returned (7, '')
2020-10-05 15:14:40,759 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:14:42 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:14:42,552 - call returned (255, '20/10/05 15:14:42 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:14:42,552 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpsozzWz 2>/tmp/tmpUt5USp''] {'quiet': False}
2020-10-05 15:14:42,612 - call returned (7, '')
2020-10-05 15:14:42,613 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:14:44 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:14:44,409 - call returned (255, '20/10/05 15:14:44 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:14:44,409 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:14:44,409 - Will retry 3 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:14:49,415 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpVvkozf 2>/tmp/tmpKhxGr_''] {'quiet': False}
2020-10-05 15:14:49,476 - call returned (7, '')
2020-10-05 15:14:49,477 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:14:51 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:14:51,313 - call returned (255, '20/10/05 15:14:51 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:14:51,314 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpxyTXId 2>/tmp/tmpH3iVvJ''] {'quiet': False}
2020-10-05 15:14:51,376 - call returned (7, '')
2020-10-05 15:14:51,376 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:14:53 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:14:53,239 - call returned (255, '20/10/05 15:14:53 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:14:53,240 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:14:53,240 - Will retry 2 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:14:58,246 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpNEFhjx 2>/tmp/tmpGQjsdH''] {'quiet': False}
2020-10-05 15:14:58,306 - call returned (7, '')
2020-10-05 15:14:58,307 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:15:00 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:15:00,124 - call returned (255, '20/10/05 15:15:00 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:15:00,125 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpqw2G_n 2>/tmp/tmp8pUyPh''] {'quiet': False}
2020-10-05 15:15:00,184 - call returned (7, '')
2020-10-05 15:15:00,185 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:15:01 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:15:01,979 - call returned (255, '20/10/05 15:15:01 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:15:01,979 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:15:01,980 - Will retry 1 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:15:06,985 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmppkKX51 2>/tmp/tmp2J7_xh''] {'quiet': False}
2020-10-05 15:15:07,045 - call returned (7, '')
2020-10-05 15:15:07,046 - Getting jmx metrics from NN failed. URL: http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmppkKX51 2>/tmp/tmp2J7_xh' returned 7. 

2020-10-05 15:15:07,046 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:15:08 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:15:08,842 - call returned (255, '20/10/05 15:15:08 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:15:08,843 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp2xVY8P 2>/tmp/tmp411OeF''] {'quiet': False}
2020-10-05 15:15:08,903 - call returned (7, '')
2020-10-05 15:15:08,903 - Getting jmx metrics from NN failed. URL: http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmp2xVY8P 2>/tmp/tmp411OeF' returned 7. 

2020-10-05 15:15:08,904 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:15:10 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:15:10,734 - call returned (255, '20/10/05 15:15:10 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:15:10,734 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:15:10,734 - No active NameNode was found after 5 retries. Will return current NameNode HA states
2020-10-05 15:15:10,734 - Will retry 120 time(s), caught exception: The NameNode nn1 is not listed as Active or Standby, waiting.... Sleeping for 5 sec(s)
2020-10-05 15:15:15,740 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpMqVKR9 2>/tmp/tmpUYxjTS''] {'quiet': False}
2020-10-05 15:15:15,805 - call returned (7, '')
2020-10-05 15:15:15,805 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:15:17 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:15:17,689 - call returned (255, '20/10/05 15:15:17 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:15:17,690 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpcDM5hh 2>/tmp/tmpy_0KSr''] {'quiet': False}
2020-10-05 15:15:17,752 - call returned (7, '')
2020-10-05 15:15:17,752 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:15:19 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:15:19,584 - call returned (255, '20/10/05 15:15:19 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:15:19,584 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:15:19,584 - Will retry 4 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:15:24,590 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpn2dNmS 2>/tmp/tmp7yFeqV''] {'quiet': False}
2020-10-05 15:15:24,649 - call returned (7, '')
2020-10-05 15:15:24,650 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:15:26 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:15:26,477 - call returned (255, '20/10/05 15:15:26 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:15:26,478 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp3VY8Fn 2>/tmp/tmp8XTkGo''] {'quiet': False}
2020-10-05 15:15:26,538 - call returned (7, '')
2020-10-05 15:15:26,538 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:15:28 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:15:28,346 - call returned (255, '20/10/05 15:15:28 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:15:28,346 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:15:28,346 - Will retry 3 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:15:33,352 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmph4YUFR 2>/tmp/tmp4bqipQ''] {'quiet': False}
2020-10-05 15:15:33,412 - call returned (7, '')
2020-10-05 15:15:33,413 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:15:35 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:15:35,209 - call returned (255, '20/10/05 15:15:35 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:15:35,209 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpcD79pv 2>/tmp/tmppg7qJt''] {'quiet': False}
2020-10-05 15:15:35,269 - call returned (7, '')
2020-10-05 15:15:35,270 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:15:37 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:15:37,080 - call returned (255, '20/10/05 15:15:37 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:15:37,080 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:15:37,080 - Will retry 2 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:15:42,086 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp1j6X9m 2>/tmp/tmpBRURVj''] {'quiet': False}
2020-10-05 15:15:42,147 - call returned (7, '')
2020-10-05 15:15:42,148 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:15:43 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:15:43,972 - call returned (255, '20/10/05 15:15:43 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:15:43,972 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpb8NtIY 2>/tmp/tmpUiVgMX''] {'quiet': False}
2020-10-05 15:15:44,034 - call returned (7, '')
2020-10-05 15:15:44,035 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:15:45 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:15:45,822 - call returned (255, '20/10/05 15:15:45 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:15:45,822 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:15:45,822 - Will retry 1 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:15:50,828 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpM_HKW1 2>/tmp/tmp8lTdb7''] {'quiet': False}
2020-10-05 15:15:50,888 - call returned (7, '')
2020-10-05 15:15:50,889 - Getting jmx metrics from NN failed. URL: http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmpM_HKW1 2>/tmp/tmp8lTdb7' returned 7. 

2020-10-05 15:15:50,889 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:15:52 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:15:52,685 - call returned (255, '20/10/05 15:15:52 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:15:52,686 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp57KnnN 2>/tmp/tmp9mFAVJ''] {'quiet': False}
2020-10-05 15:15:52,746 - call returned (7, '')
2020-10-05 15:15:52,746 - Getting jmx metrics from NN failed. URL: http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmp57KnnN 2>/tmp/tmp9mFAVJ' returned 7. 

2020-10-05 15:15:52,747 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:15:54 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:15:54,593 - call returned (255, '20/10/05 15:15:54 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:15:54,594 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:15:54,594 - No active NameNode was found after 5 retries. Will return current NameNode HA states
2020-10-05 15:15:54,594 - Will retry 119 time(s), caught exception: The NameNode nn1 is not listed as Active or Standby, waiting.... Sleeping for 5 sec(s)
2020-10-05 15:15:59,598 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpodVgHx 2>/tmp/tmpYmueUt''] {'quiet': False}
2020-10-05 15:15:59,657 - call returned (7, '')
2020-10-05 15:15:59,657 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:16:01 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:16:01,452 - call returned (255, '20/10/05 15:16:01 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:16:01,453 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpdfeTG2 2>/tmp/tmpA_rPNr''] {'quiet': False}
2020-10-05 15:16:01,513 - call returned (7, '')
2020-10-05 15:16:01,514 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:16:03 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:16:03,314 - call returned (255, '20/10/05 15:16:03 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:16:03,314 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:16:03,315 - Will retry 4 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:16:08,317 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpZk0Ox7 2>/tmp/tmpo6M68d''] {'quiet': False}
2020-10-05 15:16:08,378 - call returned (7, '')
2020-10-05 15:16:08,378 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:16:10 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:16:10,181 - call returned (255, '20/10/05 15:16:10 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:16:10,182 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp491KwU 2>/tmp/tmpzTULDH''] {'quiet': False}
2020-10-05 15:16:10,242 - call returned (7, '')
2020-10-05 15:16:10,243 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:16:12 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:16:12,127 - call returned (255, '20/10/05 15:16:12 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:16:12,127 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:16:12,127 - Will retry 3 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:16:17,133 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpxO9yiL 2>/tmp/tmppVAB0X''] {'quiet': False}
2020-10-05 15:16:17,193 - call returned (7, '')
2020-10-05 15:16:17,194 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:16:18 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:16:18,990 - call returned (255, '20/10/05 15:16:18 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:16:18,990 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpyHUa3F 2>/tmp/tmpLFrwNv''] {'quiet': False}
2020-10-05 15:16:19,051 - call returned (7, '')
2020-10-05 15:16:19,052 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:16:20 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:16:20,848 - call returned (255, '20/10/05 15:16:20 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:16:20,849 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:16:20,849 - Will retry 2 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:16:25,855 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp2j173s 2>/tmp/tmpDd1Xys''] {'quiet': False}
2020-10-05 15:16:25,914 - call returned (7, '')
2020-10-05 15:16:25,914 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:16:27 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:16:27,717 - call returned (255, '20/10/05 15:16:27 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:16:27,718 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpL_ITst 2>/tmp/tmpPwrfDD''] {'quiet': False}
2020-10-05 15:16:27,778 - call returned (7, '')
2020-10-05 15:16:27,778 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:16:29 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:16:29,575 - call returned (255, '20/10/05 15:16:29 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:16:29,575 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:16:29,575 - Will retry 1 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:16:34,581 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpTIlMrD 2>/tmp/tmpBDT05N''] {'quiet': False}
2020-10-05 15:16:34,641 - call returned (7, '')
2020-10-05 15:16:34,641 - Getting jmx metrics from NN failed. URL: http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmpTIlMrD 2>/tmp/tmpBDT05N' returned 7. 

2020-10-05 15:16:34,642 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:16:36 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:16:36,451 - call returned (255, '20/10/05 15:16:36 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:16:36,452 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpmzUzAF 2>/tmp/tmpFpLMmr''] {'quiet': False}
2020-10-05 15:16:36,512 - call returned (7, '')
2020-10-05 15:16:36,513 - Getting jmx metrics from NN failed. URL: http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmpmzUzAF 2>/tmp/tmpFpLMmr' returned 7. 

2020-10-05 15:16:36,513 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:16:38 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:16:38,338 - call returned (255, '20/10/05 15:16:38 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:16:38,339 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:16:38,339 - No active NameNode was found after 5 retries. Will return current NameNode HA states
2020-10-05 15:16:38,339 - Will retry 118 time(s), caught exception: The NameNode nn1 is not listed as Active or Standby, waiting.... Sleeping for 5 sec(s)
2020-10-05 15:16:43,345 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpxviFJd 2>/tmp/tmpIbEo2D''] {'quiet': False}
2020-10-05 15:16:43,405 - call returned (7, '')
2020-10-05 15:16:43,406 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:16:45 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:16:45,201 - call returned (255, '20/10/05 15:16:45 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:16:45,202 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpBdkEcT 2>/tmp/tmpsA0oEP''] {'quiet': False}
2020-10-05 15:16:45,262 - call returned (7, '')
2020-10-05 15:16:45,262 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:16:47 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:16:47,112 - call returned (255, '20/10/05 15:16:47 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:16:47,112 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:16:47,112 - Will retry 4 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:16:52,118 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp8gWg7u 2>/tmp/tmprFMLIV''] {'quiet': False}
2020-10-05 15:16:52,178 - call returned (7, '')
2020-10-05 15:16:52,179 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:16:53 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:16:53,984 - call returned (255, '20/10/05 15:16:53 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:16:53,984 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpecSYiU 2>/tmp/tmpKP8vc9''] {'quiet': False}
2020-10-05 15:16:54,047 - call returned (7, '')
2020-10-05 15:16:54,048 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:16:55 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:16:55,878 - call returned (255, '20/10/05 15:16:55 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:16:55,878 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:16:55,878 - Will retry 3 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:17:00,882 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpzoI2EX 2>/tmp/tmpyhDDNW''] {'quiet': False}
2020-10-05 15:17:00,942 - call returned (7, '')
2020-10-05 15:17:00,942 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:17:02 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:17:02,732 - call returned (255, '20/10/05 15:17:02 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:17:02,732 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpCjTZUw 2>/tmp/tmpMD9eRZ''] {'quiet': False}
2020-10-05 15:17:02,793 - call returned (7, '')
2020-10-05 15:17:02,793 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:17:04 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:17:04,635 - call returned (255, '20/10/05 15:17:04 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:17:04,635 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:17:04,635 - Will retry 2 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:17:09,641 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp8Lb0p0 2>/tmp/tmpzC6Ge5''] {'quiet': False}
2020-10-05 15:17:09,703 - call returned (7, '')
2020-10-05 15:17:09,704 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:17:11 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:17:11,505 - call returned (255, '20/10/05 15:17:11 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:17:11,506 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp5cudDF 2>/tmp/tmpsUW59D''] {'quiet': False}
2020-10-05 15:17:11,566 - call returned (7, '')
2020-10-05 15:17:11,567 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:17:13 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:17:13,409 - call returned (255, '20/10/05 15:17:13 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:17:13,409 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:17:13,409 - Will retry 1 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:17:18,411 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpfLwQP0 2>/tmp/tmptfA0cY''] {'quiet': False}
2020-10-05 15:17:18,470 - call returned (7, '')
2020-10-05 15:17:18,471 - Getting jmx metrics from NN failed. URL: http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmpfLwQP0 2>/tmp/tmptfA0cY' returned 7. 

2020-10-05 15:17:18,471 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:17:20 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:17:20,357 - call returned (255, '20/10/05 15:17:20 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:17:20,358 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpfmCRH3 2>/tmp/tmpQ1m48b''] {'quiet': False}
2020-10-05 15:17:20,420 - call returned (7, '')
2020-10-05 15:17:20,420 - Getting jmx metrics from NN failed. URL: http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmpfmCRH3 2>/tmp/tmpQ1m48b' returned 7. 

2020-10-05 15:17:20,421 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:17:22 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:17:22,250 - call returned (255, '20/10/05 15:17:22 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:17:22,250 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:17:22,250 - No active NameNode was found after 5 retries. Will return current NameNode HA states
2020-10-05 15:17:22,251 - Will retry 117 time(s), caught exception: The NameNode nn1 is not listed as Active or Standby, waiting.... Sleeping for 5 sec(s)
2020-10-05 15:17:27,256 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpZerKha 2>/tmp/tmp4RclZM''] {'quiet': False}
2020-10-05 15:17:27,317 - call returned (7, '')
2020-10-05 15:17:27,317 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:17:29 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:17:29,118 - call returned (255, '20/10/05 15:17:29 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:17:29,118 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpfKoHg3 2>/tmp/tmpWvHU4M''] {'quiet': False}
2020-10-05 15:17:29,178 - call returned (7, '')
2020-10-05 15:17:29,179 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:17:30 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:17:30,974 - call returned (255, '20/10/05 15:17:30 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:17:30,975 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:17:30,975 - Will retry 4 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:17:35,980 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpQSObF1 2>/tmp/tmpAUXEbt''] {'quiet': False}
2020-10-05 15:17:36,040 - call returned (7, '')
2020-10-05 15:17:36,041 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:17:37 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:17:37,836 - call returned (255, '20/10/05 15:17:37 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:17:37,837 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpCnih3v 2>/tmp/tmpakQiGi''] {'quiet': False}
2020-10-05 15:17:37,897 - call returned (7, '')
2020-10-05 15:17:37,898 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:17:39 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:17:39,695 - call returned (255, '20/10/05 15:17:39 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:17:39,695 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:17:39,695 - Will retry 3 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:17:44,701 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpJ4wBbj 2>/tmp/tmp0qZnGj''] {'quiet': False}
2020-10-05 15:17:44,763 - call returned (7, '')
2020-10-05 15:17:44,764 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:17:46 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:17:46,591 - call returned (255, '20/10/05 15:17:46 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:17:46,592 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp8ld6Ls 2>/tmp/tmphQ45jj''] {'quiet': False}
2020-10-05 15:17:46,652 - call returned (7, '')
2020-10-05 15:17:46,653 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:17:48 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:17:48,485 - call returned (255, '20/10/05 15:17:48 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:17:48,485 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:17:48,485 - Will retry 2 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:17:53,491 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp4qMn7P 2>/tmp/tmp8ybrkG''] {'quiet': False}
2020-10-05 15:17:53,551 - call returned (7, '')
2020-10-05 15:17:53,552 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:17:55 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:17:55,351 - call returned (255, '20/10/05 15:17:55 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:17:55,351 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpUOcFkg 2>/tmp/tmpJ9JtfE''] {'quiet': False}
2020-10-05 15:17:55,411 - call returned (7, '')
2020-10-05 15:17:55,412 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:17:57 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:17:57,218 - call returned (255, '20/10/05 15:17:57 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:17:57,218 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:17:57,219 - Will retry 1 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:18:02,224 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpsqkraN 2>/tmp/tmpdSJQcw''] {'quiet': False}
2020-10-05 15:18:02,284 - call returned (7, '')
2020-10-05 15:18:02,284 - Getting jmx metrics from NN failed. URL: http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmpsqkraN 2>/tmp/tmpdSJQcw' returned 7. 

2020-10-05 15:18:02,285 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:18:04 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:18:04,088 - call returned (255, '20/10/05 15:18:04 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:18:04,089 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp9ZWvel 2>/tmp/tmpeBuUav''] {'quiet': False}
2020-10-05 15:18:04,149 - call returned (7, '')
2020-10-05 15:18:04,150 - Getting jmx metrics from NN failed. URL: http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmp9ZWvel 2>/tmp/tmpeBuUav' returned 7. 

2020-10-05 15:18:04,150 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:18:05 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:18:05,948 - call returned (255, '20/10/05 15:18:05 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:18:05,948 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:18:05,948 - No active NameNode was found after 5 retries. Will return current NameNode HA states
2020-10-05 15:18:05,948 - Will retry 116 time(s), caught exception: The NameNode nn1 is not listed as Active or Standby, waiting.... Sleeping for 5 sec(s)
2020-10-05 15:18:10,951 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpQvkuZu 2>/tmp/tmpDO0mIQ''] {'quiet': False}
2020-10-05 15:18:11,014 - call returned (7, '')
2020-10-05 15:18:11,015 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:18:12 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:18:12,930 - call returned (255, '20/10/05 15:18:12 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:18:12,931 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpxyLgIB 2>/tmp/tmpY2nSBD''] {'quiet': False}
2020-10-05 15:18:12,993 - call returned (7, '')
2020-10-05 15:18:12,993 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:18:14 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:18:14,810 - call returned (255, '20/10/05 15:18:14 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:18:14,810 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:18:14,811 - Will retry 4 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:18:19,816 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp8fESUY 2>/tmp/tmpi0Jcon''] {'quiet': False}
2020-10-05 15:18:19,876 - call returned (7, '')
2020-10-05 15:18:19,877 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:18:21 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:18:21,677 - call returned (255, '20/10/05 15:18:21 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:18:21,677 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpLi6KZo 2>/tmp/tmpHXjbW3''] {'quiet': False}
2020-10-05 15:18:21,737 - call returned (7, '')
2020-10-05 15:18:21,737 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:18:23 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:18:23,534 - call returned (255, '20/10/05 15:18:23 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:18:23,535 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:18:23,535 - Will retry 3 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:18:28,540 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpzFPZNH 2>/tmp/tmpTvRUjx''] {'quiet': False}
2020-10-05 15:18:28,600 - call returned (7, '')
2020-10-05 15:18:28,600 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:18:30 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:18:30,396 - call returned (255, '20/10/05 15:18:30 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:18:30,396 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpF5ukZ1 2>/tmp/tmpNVrgcF''] {'quiet': False}
2020-10-05 15:18:30,456 - call returned (7, '')
2020-10-05 15:18:30,457 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:18:32 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:18:32,251 - call returned (255, '20/10/05 15:18:32 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:18:32,251 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:18:32,252 - Will retry 2 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:18:37,257 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpyFnFPf 2>/tmp/tmpmsHgzh''] {'quiet': False}
2020-10-05 15:18:37,325 - call returned (7, '')
2020-10-05 15:18:37,325 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:18:39 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:18:39,167 - call returned (255, '20/10/05 15:18:39 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:18:39,167 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpl1UoJN 2>/tmp/tmpVr3Mii''] {'quiet': False}
2020-10-05 15:18:39,237 - call returned (7, '')
2020-10-05 15:18:39,238 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:18:41 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:18:41,079 - call returned (255, '20/10/05 15:18:41 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:18:41,079 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:18:41,079 - Will retry 1 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:18:46,082 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpWqhhPZ 2>/tmp/tmpMEb4u_''] {'quiet': False}
2020-10-05 15:18:46,141 - call returned (7, '')
2020-10-05 15:18:46,142 - Getting jmx metrics from NN failed. URL: http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmpWqhhPZ 2>/tmp/tmpMEb4u_' returned 7. 

2020-10-05 15:18:46,142 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:18:47 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:18:47,948 - call returned (255, '20/10/05 15:18:47 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:18:47,949 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpOXd_xb 2>/tmp/tmpRAECcf''] {'quiet': False}
2020-10-05 15:18:48,009 - call returned (7, '')
2020-10-05 15:18:48,010 - Getting jmx metrics from NN failed. URL: http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmpOXd_xb 2>/tmp/tmpRAECcf' returned 7. 

2020-10-05 15:18:48,010 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:18:49 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:18:49,815 - call returned (255, '20/10/05 15:18:49 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:18:49,816 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:18:49,816 - No active NameNode was found after 5 retries. Will return current NameNode HA states
2020-10-05 15:18:49,816 - Will retry 115 time(s), caught exception: The NameNode nn1 is not listed as Active or Standby, waiting.... Sleeping for 5 sec(s)
2020-10-05 15:18:54,822 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp2LH0TT 2>/tmp/tmpOR12xG''] {'quiet': False}
2020-10-05 15:18:54,882 - call returned (7, '')
2020-10-05 15:18:54,883 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:18:56 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:18:56,683 - call returned (255, '20/10/05 15:18:56 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:18:56,684 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp_aQBg_ 2>/tmp/tmpAFWC8f''] {'quiet': False}
2020-10-05 15:18:56,744 - call returned (7, '')
2020-10-05 15:18:56,744 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:18:58 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:18:58,559 - call returned (255, '20/10/05 15:18:58 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:18:58,560 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:18:58,560 - Will retry 4 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:19:03,562 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpgUfPSL 2>/tmp/tmp3RyoY7''] {'quiet': False}
2020-10-05 15:19:03,623 - call returned (7, '')
2020-10-05 15:19:03,624 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:19:05 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:19:05,452 - call returned (255, '20/10/05 15:19:05 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:19:05,453 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmphLLBul 2>/tmp/tmpqDjI4T''] {'quiet': False}
2020-10-05 15:19:05,513 - call returned (7, '')
2020-10-05 15:19:05,514 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:19:07 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:19:07,318 - call returned (255, '20/10/05 15:19:07 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:19:07,319 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:19:07,319 - Will retry 3 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:19:12,324 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpVzFFvD 2>/tmp/tmp_zAlp5''] {'quiet': False}
2020-10-05 15:19:12,384 - call returned (7, '')
2020-10-05 15:19:12,384 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:19:14 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:19:14,204 - call returned (255, '20/10/05 15:19:14 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:19:14,204 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpT5hySC 2>/tmp/tmpH7k1eX''] {'quiet': False}
2020-10-05 15:19:14,264 - call returned (7, '')
2020-10-05 15:19:14,265 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:19:16 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:19:16,065 - call returned (255, '20/10/05 15:19:16 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:19:16,065 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:19:16,065 - Will retry 2 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:19:21,070 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp_8cGXF 2>/tmp/tmpSTop8B''] {'quiet': False}
2020-10-05 15:19:21,130 - call returned (7, '')
2020-10-05 15:19:21,131 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:19:22 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:19:22,940 - call returned (255, '20/10/05 15:19:22 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:19:22,941 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpmwBc27 2>/tmp/tmp3GSzsd''] {'quiet': False}
2020-10-05 15:19:23,001 - call returned (7, '')
2020-10-05 15:19:23,002 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:19:24 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:19:24,855 - call returned (255, '20/10/05 15:19:24 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:19:24,855 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:19:24,855 - Will retry 1 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:19:29,861 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp2ahEp7 2>/tmp/tmptGjscd''] {'quiet': False}
2020-10-05 15:19:29,921 - call returned (7, '')
2020-10-05 15:19:29,922 - Getting jmx metrics from NN failed. URL: http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmp2ahEp7 2>/tmp/tmptGjscd' returned 7. 

2020-10-05 15:19:29,922 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:19:31 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:19:31,735 - call returned (255, '20/10/05 15:19:31 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:19:31,736 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp8V0oze 2>/tmp/tmp5qtfG4''] {'quiet': False}
2020-10-05 15:19:31,797 - call returned (7, '')
2020-10-05 15:19:31,797 - Getting jmx metrics from NN failed. URL: http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmp8V0oze 2>/tmp/tmp5qtfG4' returned 7. 

2020-10-05 15:19:31,798 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:19:33 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:19:33,610 - call returned (255, '20/10/05 15:19:33 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:19:33,610 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:19:33,610 - No active NameNode was found after 5 retries. Will return current NameNode HA states
2020-10-05 15:19:33,611 - Will retry 114 time(s), caught exception: The NameNode nn1 is not listed as Active or Standby, waiting.... Sleeping for 5 sec(s)
2020-10-05 15:19:38,616 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp1R2ydM 2>/tmp/tmpBo15yr''] {'quiet': False}
2020-10-05 15:19:38,676 - call returned (7, '')
2020-10-05 15:19:38,677 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:19:40 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:19:40,528 - call returned (255, '20/10/05 15:19:40 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:19:40,529 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpRZ48vg 2>/tmp/tmpd06Wn6''] {'quiet': False}
2020-10-05 15:19:40,600 - call returned (7, '')
2020-10-05 15:19:40,601 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:19:42 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:19:42,557 - call returned (255, '20/10/05 15:19:42 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:19:42,557 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:19:42,557 - Will retry 4 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:19:47,563 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp2uEmzR 2>/tmp/tmp_0QaAT''] {'quiet': False}
2020-10-05 15:19:47,623 - call returned (7, '')
2020-10-05 15:19:47,624 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:19:49 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:19:49,425 - call returned (255, '20/10/05 15:19:49 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:19:49,426 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpqtX7se 2>/tmp/tmpVPREFU''] {'quiet': False}
2020-10-05 15:19:49,489 - call returned (7, '')
2020-10-05 15:19:49,489 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:19:51 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:19:51,319 - call returned (255, '20/10/05 15:19:51 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:19:51,319 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:19:51,319 - Will retry 3 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:19:56,325 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpXCcV4a 2>/tmp/tmpcGOFLX''] {'quiet': False}
2020-10-05 15:19:56,385 - call returned (7, '')
2020-10-05 15:19:56,386 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:19:58 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:19:58,183 - call returned (255, '20/10/05 15:19:58 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:19:58,184 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpuRW6EE 2>/tmp/tmp4KhOtH''] {'quiet': False}
2020-10-05 15:19:58,244 - call returned (7, '')
2020-10-05 15:19:58,245 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:20:00 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:20:00,075 - call returned (255, '20/10/05 15:20:00 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:20:00,075 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:20:00,075 - Will retry 2 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:20:05,081 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpaOHT5B 2>/tmp/tmpy1FzZ4''] {'quiet': False}
2020-10-05 15:20:05,140 - call returned (7, '')
2020-10-05 15:20:05,141 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:20:06 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:20:06,984 - call returned (255, '20/10/05 15:20:06 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:20:06,985 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpsiL8jp 2>/tmp/tmpO5Ufp_''] {'quiet': False}
2020-10-05 15:20:07,045 - call returned (7, '')
2020-10-05 15:20:07,046 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:20:08 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:20:08,841 - call returned (255, '20/10/05 15:20:08 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:20:08,842 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:20:08,842 - Will retry 1 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:20:13,847 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpJjaPWZ 2>/tmp/tmpqmsa26''] {'quiet': False}
2020-10-05 15:20:13,908 - call returned (7, '')
2020-10-05 15:20:13,908 - Getting jmx metrics from NN failed. URL: http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmpJjaPWZ 2>/tmp/tmpqmsa26' returned 7. 

2020-10-05 15:20:13,909 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:20:15 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:20:15,733 - call returned (255, '20/10/05 15:20:15 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:20:15,733 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpYkYlGD 2>/tmp/tmpzxc2aG''] {'quiet': False}
2020-10-05 15:20:15,796 - call returned (7, '')
2020-10-05 15:20:15,797 - Getting jmx metrics from NN failed. URL: http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmpYkYlGD 2>/tmp/tmpzxc2aG' returned 7. 

2020-10-05 15:20:15,797 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:20:17 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:20:17,636 - call returned (255, '20/10/05 15:20:17 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:20:17,636 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:20:17,636 - No active NameNode was found after 5 retries. Will return current NameNode HA states
2020-10-05 15:20:17,636 - Will retry 113 time(s), caught exception: The NameNode nn1 is not listed as Active or Standby, waiting.... Sleeping for 5 sec(s)
2020-10-05 15:20:22,642 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpqRz8HA 2>/tmp/tmp9hoCSR''] {'quiet': False}
2020-10-05 15:20:22,702 - call returned (7, '')
2020-10-05 15:20:22,703 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:20:24 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:20:24,509 - call returned (255, '20/10/05 15:20:24 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:20:24,509 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpYwo4ot 2>/tmp/tmppNiWgW''] {'quiet': False}
2020-10-05 15:20:24,569 - call returned (7, '')
2020-10-05 15:20:24,570 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:20:26 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:20:26,362 - call returned (255, '20/10/05 15:20:26 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:20:26,363 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:20:26,363 - Will retry 4 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:20:31,368 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmplIukZc 2>/tmp/tmpqHxW1n''] {'quiet': False}
2020-10-05 15:20:31,428 - call returned (7, '')
2020-10-05 15:20:31,429 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:20:33 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:20:33,281 - call returned (255, '20/10/05 15:20:33 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:20:33,282 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpx2NLQE 2>/tmp/tmpUkRLhk''] {'quiet': False}
2020-10-05 15:20:33,342 - call returned (7, '')
2020-10-05 15:20:33,342 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:20:35 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:20:35,148 - call returned (255, '20/10/05 15:20:35 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:20:35,148 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:20:35,148 - Will retry 3 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:20:40,154 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpHUOHgN 2>/tmp/tmpBfInSe''] {'quiet': False}
2020-10-05 15:20:40,215 - call returned (7, '')
2020-10-05 15:20:40,216 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:20:42 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:20:42,093 - call returned (255, '20/10/05 15:20:42 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:20:42,094 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpPWkBWi 2>/tmp/tmp1QTDA4''] {'quiet': False}
2020-10-05 15:20:42,161 - call returned (7, '')
2020-10-05 15:20:42,162 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:20:43 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:20:43,988 - call returned (255, '20/10/05 15:20:43 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:20:43,989 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:20:43,989 - Will retry 2 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:20:48,990 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpDFsoqo 2>/tmp/tmpTsVniK''] {'quiet': False}
2020-10-05 15:20:49,050 - call returned (7, '')
2020-10-05 15:20:49,051 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:20:50 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:20:50,853 - call returned (255, '20/10/05 15:20:50 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:20:50,854 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpke_TY6 2>/tmp/tmpe7J839''] {'quiet': False}
2020-10-05 15:20:50,914 - call returned (7, '')
2020-10-05 15:20:50,915 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:20:52 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:20:52,719 - call returned (255, '20/10/05 15:20:52 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:20:52,719 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:20:52,720 - Will retry 1 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:20:57,725 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpKTlwC_ 2>/tmp/tmpmL_0XT''] {'quiet': False}
2020-10-05 15:20:57,785 - call returned (7, '')
2020-10-05 15:20:57,785 - Getting jmx metrics from NN failed. URL: http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmpKTlwC_ 2>/tmp/tmpmL_0XT' returned 7. 

2020-10-05 15:20:57,785 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:20:59 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:20:59,585 - call returned (255, '20/10/05 15:20:59 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:20:59,586 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpRPMn8Z 2>/tmp/tmpHRuhXs''] {'quiet': False}
2020-10-05 15:20:59,646 - call returned (7, '')
2020-10-05 15:20:59,647 - Getting jmx metrics from NN failed. URL: http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmpRPMn8Z 2>/tmp/tmpHRuhXs' returned 7. 

2020-10-05 15:20:59,647 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:21:01 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:21:01,516 - call returned (255, '20/10/05 15:21:01 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:21:01,516 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:21:01,516 - No active NameNode was found after 5 retries. Will return current NameNode HA states
2020-10-05 15:21:01,516 - Will retry 112 time(s), caught exception: The NameNode nn1 is not listed as Active or Standby, waiting.... Sleeping for 5 sec(s)
2020-10-05 15:21:06,522 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpwfaMoF 2>/tmp/tmpBR9H8o''] {'quiet': False}
2020-10-05 15:21:06,583 - call returned (7, '')
2020-10-05 15:21:06,584 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:21:08 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:21:08,394 - call returned (255, '20/10/05 15:21:08 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:21:08,394 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpkYX0w4 2>/tmp/tmpr8H1NI''] {'quiet': False}
2020-10-05 15:21:08,458 - call returned (7, '')
2020-10-05 15:21:08,458 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:21:10 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:21:10,276 - call returned (255, '20/10/05 15:21:10 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:21:10,276 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:21:10,276 - Will retry 4 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:21:15,280 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpf2q2g3 2>/tmp/tmpXONRqN''] {'quiet': False}
2020-10-05 15:21:15,340 - call returned (7, '')
2020-10-05 15:21:15,341 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:21:17 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:21:17,136 - call returned (255, '20/10/05 15:21:17 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:21:17,137 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp7KrmHL 2>/tmp/tmpya576m''] {'quiet': False}
2020-10-05 15:21:17,198 - call returned (7, '')
2020-10-05 15:21:17,198 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:21:19 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:21:19,035 - call returned (255, '20/10/05 15:21:19 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:21:19,035 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:21:19,035 - Will retry 3 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:21:24,041 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpBcX6NX 2>/tmp/tmpV04l7s''] {'quiet': False}
2020-10-05 15:21:24,101 - call returned (7, '')
2020-10-05 15:21:24,102 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:21:25 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:21:25,901 - call returned (255, '20/10/05 15:21:25 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:21:25,902 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpO5SDcI 2>/tmp/tmp3GEhoC''] {'quiet': False}
2020-10-05 15:21:25,962 - call returned (7, '')
2020-10-05 15:21:25,963 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:21:27 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:21:27,760 - call returned (255, '20/10/05 15:21:27 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:21:27,760 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:21:27,760 - Will retry 2 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:21:32,762 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmprQKl1r 2>/tmp/tmpepKs9W''] {'quiet': False}
2020-10-05 15:21:32,823 - call returned (7, '')
2020-10-05 15:21:32,824 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:21:34 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:21:34,656 - call returned (255, '20/10/05 15:21:34 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:21:34,656 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpcNFcER 2>/tmp/tmpFXOEOl''] {'quiet': False}
2020-10-05 15:21:34,716 - call returned (7, '')
2020-10-05 15:21:34,717 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:21:36 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:21:36,513 - call returned (255, '20/10/05 15:21:36 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:21:36,513 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:21:36,513 - Will retry 1 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:21:41,515 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpal3yWi 2>/tmp/tmpjNBL3D''] {'quiet': False}
2020-10-05 15:21:41,574 - call returned (7, '')
2020-10-05 15:21:41,575 - Getting jmx metrics from NN failed. URL: http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmpal3yWi 2>/tmp/tmpjNBL3D' returned 7. 

2020-10-05 15:21:41,575 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:21:43 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:21:43,377 - call returned (255, '20/10/05 15:21:43 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:21:43,378 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpUg1Yve 2>/tmp/tmpIFSnw3''] {'quiet': False}
2020-10-05 15:21:43,437 - call returned (7, '')
2020-10-05 15:21:43,438 - Getting jmx metrics from NN failed. URL: http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmpUg1Yve 2>/tmp/tmpIFSnw3' returned 7. 

2020-10-05 15:21:43,438 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:21:45 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:21:45,239 - call returned (255, '20/10/05 15:21:45 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:21:45,239 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:21:45,239 - No active NameNode was found after 5 retries. Will return current NameNode HA states
2020-10-05 15:21:45,240 - Will retry 111 time(s), caught exception: The NameNode nn1 is not listed as Active or Standby, waiting.... Sleeping for 5 sec(s)
2020-10-05 15:21:50,245 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpVNnge0 2>/tmp/tmpVzupW3''] {'quiet': False}
2020-10-05 15:21:50,305 - call returned (7, '')
2020-10-05 15:21:50,305 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:21:52 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:21:52,102 - call returned (255, '20/10/05 15:21:52 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:21:52,102 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpzDWspX 2>/tmp/tmpJ9Pk1C''] {'quiet': False}
2020-10-05 15:21:52,162 - call returned (7, '')
2020-10-05 15:21:52,163 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:21:53 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:21:54,002 - call returned (255, '20/10/05 15:21:53 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:21:54,002 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:21:54,002 - Will retry 4 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:21:59,006 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpx8Bzod 2>/tmp/tmpGQU6AJ''] {'quiet': False}
2020-10-05 15:21:59,066 - call returned (7, '')
2020-10-05 15:21:59,067 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:22:00 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:22:00,882 - call returned (255, '20/10/05 15:22:00 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:22:00,883 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp7gZ4mc 2>/tmp/tmprpx50Z''] {'quiet': False}
2020-10-05 15:22:00,944 - call returned (7, '')
2020-10-05 15:22:00,944 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:22:02 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:22:02,802 - call returned (255, '20/10/05 15:22:02 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:22:02,802 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:22:02,802 - Will retry 3 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:22:07,808 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpcapBXa 2>/tmp/tmpmK2qZI''] {'quiet': False}
2020-10-05 15:22:07,868 - call returned (7, '')
2020-10-05 15:22:07,869 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:22:09 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:22:09,666 - call returned (255, '20/10/05 15:22:09 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:22:09,667 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpYHsMvr 2>/tmp/tmp6Djo0C''] {'quiet': False}
2020-10-05 15:22:09,726 - call returned (7, '')
2020-10-05 15:22:09,727 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:22:11 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:22:11,559 - call returned (255, '20/10/05 15:22:11 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:22:11,560 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:22:11,560 - Will retry 2 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:22:16,566 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpGCca3l 2>/tmp/tmp82o2xs''] {'quiet': False}
2020-10-05 15:22:16,626 - call returned (7, '')
2020-10-05 15:22:16,626 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:22:18 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:22:18,421 - call returned (255, '20/10/05 15:22:18 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:22:18,421 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpmU8B4g 2>/tmp/tmpSYjMit''] {'quiet': False}
2020-10-05 15:22:18,481 - call returned (7, '')
2020-10-05 15:22:18,482 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:22:20 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:22:20,307 - call returned (255, '20/10/05 15:22:20 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:22:20,307 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:22:20,307 - Will retry 1 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:22:25,313 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpH6KVvz 2>/tmp/tmpuLJLlY''] {'quiet': False}
2020-10-05 15:22:25,373 - call returned (7, '')
2020-10-05 15:22:25,374 - Getting jmx metrics from NN failed. URL: http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmpH6KVvz 2>/tmp/tmpuLJLlY' returned 7. 

2020-10-05 15:22:25,374 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:22:27 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:22:27,172 - call returned (255, '20/10/05 15:22:27 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:22:27,173 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpPDFpiw 2>/tmp/tmpQgD992''] {'quiet': False}
2020-10-05 15:22:27,234 - call returned (7, '')
2020-10-05 15:22:27,235 - Getting jmx metrics from NN failed. URL: http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmpPDFpiw 2>/tmp/tmpQgD992' returned 7. 

2020-10-05 15:22:27,235 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:22:29 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:22:29,035 - call returned (255, '20/10/05 15:22:29 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:22:29,035 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:22:29,035 - No active NameNode was found after 5 retries. Will return current NameNode HA states
2020-10-05 15:22:29,035 - Will retry 110 time(s), caught exception: The NameNode nn1 is not listed as Active or Standby, waiting.... Sleeping for 5 sec(s)
2020-10-05 15:22:34,041 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpGyoMIE 2>/tmp/tmpeiUUFQ''] {'quiet': False}
2020-10-05 15:22:34,101 - call returned (7, '')
2020-10-05 15:22:34,101 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:22:35 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:22:35,942 - call returned (255, '20/10/05 15:22:35 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:22:35,943 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpPvU9tG 2>/tmp/tmpJ_V8ge''] {'quiet': False}
2020-10-05 15:22:36,002 - call returned (7, '')
2020-10-05 15:22:36,003 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:22:37 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:22:37,799 - call returned (255, '20/10/05 15:22:37 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:22:37,799 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:22:37,800 - Will retry 4 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:22:42,804 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpQqwaKV 2>/tmp/tmpp4t17K''] {'quiet': False}
2020-10-05 15:22:42,867 - call returned (7, '')
2020-10-05 15:22:42,868 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:22:44 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:22:44,725 - call returned (255, '20/10/05 15:22:44 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:22:44,726 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpDvAHGC 2>/tmp/tmpbSwzvO''] {'quiet': False}
2020-10-05 15:22:44,787 - call returned (7, '')
2020-10-05 15:22:44,787 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:22:46 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:22:46,610 - call returned (255, '20/10/05 15:22:46 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:22:46,610 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:22:46,610 - Will retry 3 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:22:51,616 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpPFZC07 2>/tmp/tmpUNcs4r''] {'quiet': False}
2020-10-05 15:22:51,676 - call returned (7, '')
2020-10-05 15:22:51,677 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:22:53 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:22:53,480 - call returned (255, '20/10/05 15:22:53 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:22:53,480 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpLJw0Br 2>/tmp/tmpHdGTr8''] {'quiet': False}
2020-10-05 15:22:53,540 - call returned (7, '')
2020-10-05 15:22:53,541 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:22:55 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:22:55,337 - call returned (255, '20/10/05 15:22:55 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:22:55,337 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:22:55,337 - Will retry 2 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:23:00,343 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpwX5666 2>/tmp/tmpkjE6IP''] {'quiet': False}
2020-10-05 15:23:00,402 - call returned (7, '')
2020-10-05 15:23:00,403 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:23:02 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:23:02,199 - call returned (255, '20/10/05 15:23:02 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:23:02,200 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp3MU_wT 2>/tmp/tmpouxFVA''] {'quiet': False}
2020-10-05 15:23:02,259 - call returned (7, '')
2020-10-05 15:23:02,260 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:23:04 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:23:04,067 - call returned (255, '20/10/05 15:23:04 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:23:04,067 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:23:04,067 - Will retry 1 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:23:09,073 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpvuUdJO 2>/tmp/tmpMvVhUt''] {'quiet': False}
2020-10-05 15:23:09,135 - call returned (7, '')
2020-10-05 15:23:09,136 - Getting jmx metrics from NN failed. URL: http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmpvuUdJO 2>/tmp/tmpMvVhUt' returned 7. 

2020-10-05 15:23:09,136 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:23:10 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:23:10,963 - call returned (255, '20/10/05 15:23:10 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:23:10,964 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpyUW6cj 2>/tmp/tmpzL4dN5''] {'quiet': False}
2020-10-05 15:23:11,027 - call returned (7, '')
2020-10-05 15:23:11,027 - Getting jmx metrics from NN failed. URL: http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmpyUW6cj 2>/tmp/tmpzL4dN5' returned 7. 

2020-10-05 15:23:11,027 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:23:12 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:23:12,903 - call returned (255, '20/10/05 15:23:12 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:23:12,903 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:23:12,903 - No active NameNode was found after 5 retries. Will return current NameNode HA states
2020-10-05 15:23:12,904 - Will retry 109 time(s), caught exception: The NameNode nn1 is not listed as Active or Standby, waiting.... Sleeping for 5 sec(s)
2020-10-05 15:23:17,909 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmprEl9rK 2>/tmp/tmpJtbwcg''] {'quiet': False}
2020-10-05 15:23:17,969 - call returned (7, '')
2020-10-05 15:23:17,970 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:23:19 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:23:19,769 - call returned (255, '20/10/05 15:23:19 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:23:19,770 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpw_Ujbd 2>/tmp/tmpiJcQsJ''] {'quiet': False}
2020-10-05 15:23:19,830 - call returned (7, '')
2020-10-05 15:23:19,830 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:23:21 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:23:21,630 - call returned (255, '20/10/05 15:23:21 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:23:21,630 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:23:21,630 - Will retry 4 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:23:26,633 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpvqwCMm 2>/tmp/tmpB5BowI''] {'quiet': False}
2020-10-05 15:23:26,693 - call returned (7, '')
2020-10-05 15:23:26,693 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:23:28 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:23:28,485 - call returned (255, '20/10/05 15:23:28 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:23:28,485 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpCf3MKg 2>/tmp/tmpmhi7PI''] {'quiet': False}
2020-10-05 15:23:28,545 - call returned (7, '')
2020-10-05 15:23:28,546 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:23:30 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:23:30,345 - call returned (255, '20/10/05 15:23:30 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:23:30,345 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:23:30,345 - Will retry 3 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:23:35,346 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpP2LEsY 2>/tmp/tmp9mDzDn''] {'quiet': False}
2020-10-05 15:23:35,407 - call returned (7, '')
2020-10-05 15:23:35,408 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:23:37 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:23:37,253 - call returned (255, '20/10/05 15:23:37 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:23:37,254 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpIahOgh 2>/tmp/tmp1_BdCH''] {'quiet': False}
2020-10-05 15:23:37,315 - call returned (7, '')
2020-10-05 15:23:37,316 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:23:39 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:23:39,139 - call returned (255, '20/10/05 15:23:39 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:23:39,139 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:23:39,139 - Will retry 2 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:23:44,145 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpO5cfMt 2>/tmp/tmp_HChsx''] {'quiet': False}
2020-10-05 15:23:44,205 - call returned (7, '')
2020-10-05 15:23:44,205 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:23:46 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:23:46,082 - call returned (255, '20/10/05 15:23:46 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:23:46,082 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp8tFOLT 2>/tmp/tmpXW4ZQC''] {'quiet': False}
2020-10-05 15:23:46,144 - call returned (7, '')
2020-10-05 15:23:46,144 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:23:47 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:23:47,942 - call returned (255, '20/10/05 15:23:47 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:23:47,943 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:23:47,943 - Will retry 1 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:23:52,949 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp_lR22X 2>/tmp/tmpW07sND''] {'quiet': False}
2020-10-05 15:23:53,009 - call returned (7, '')
2020-10-05 15:23:53,009 - Getting jmx metrics from NN failed. URL: http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmp_lR22X 2>/tmp/tmpW07sND' returned 7. 

2020-10-05 15:23:53,010 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:23:54 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:23:54,815 - call returned (255, '20/10/05 15:23:54 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:23:54,815 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmprr9HEF 2>/tmp/tmpDzOXdN''] {'quiet': False}
2020-10-05 15:23:54,875 - call returned (7, '')
2020-10-05 15:23:54,876 - Getting jmx metrics from NN failed. URL: http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmprr9HEF 2>/tmp/tmpDzOXdN' returned 7. 

2020-10-05 15:23:54,876 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:23:56 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:23:56,680 - call returned (255, '20/10/05 15:23:56 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:23:56,680 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:23:56,680 - No active NameNode was found after 5 retries. Will return current NameNode HA states
2020-10-05 15:23:56,681 - Will retry 108 time(s), caught exception: The NameNode nn1 is not listed as Active or Standby, waiting.... Sleeping for 5 sec(s)
2020-10-05 15:24:01,686 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp_D1jRG 2>/tmp/tmph62tB0''] {'quiet': False}
2020-10-05 15:24:01,746 - call returned (7, '')
2020-10-05 15:24:01,747 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:24:03 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:24:03,582 - call returned (255, '20/10/05 15:24:03 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:24:03,583 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmperEzV6 2>/tmp/tmpMTVnRz''] {'quiet': False}
2020-10-05 15:24:03,646 - call returned (7, '')
2020-10-05 15:24:03,646 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:24:05 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:24:05,457 - call returned (255, '20/10/05 15:24:05 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:24:05,458 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:24:05,458 - Will retry 4 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:24:10,463 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpXki8LX 2>/tmp/tmpLMwgVx''] {'quiet': False}
2020-10-05 15:24:10,523 - call returned (7, '')
2020-10-05 15:24:10,524 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:24:12 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:24:12,336 - call returned (255, '20/10/05 15:24:12 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:24:12,337 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpRLVfwl 2>/tmp/tmp8WiX47''] {'quiet': False}
2020-10-05 15:24:12,397 - call returned (7, '')
2020-10-05 15:24:12,398 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:24:14 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:24:14,197 - call returned (255, '20/10/05 15:24:14 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:24:14,197 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:24:14,197 - Will retry 3 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:24:19,203 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpiFqHl0 2>/tmp/tmp8O05jI''] {'quiet': False}
2020-10-05 15:24:19,263 - call returned (7, '')
2020-10-05 15:24:19,263 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:24:21 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:24:21,057 - call returned (255, '20/10/05 15:24:21 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:24:21,057 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpg2NOu5 2>/tmp/tmp0zO81x''] {'quiet': False}
2020-10-05 15:24:21,117 - call returned (7, '')
2020-10-05 15:24:21,118 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:24:22 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:24:22,917 - call returned (255, '20/10/05 15:24:22 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:24:22,917 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:24:22,917 - Will retry 2 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:24:27,920 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpigh015 2>/tmp/tmp2io9ao''] {'quiet': False}
2020-10-05 15:24:27,981 - call returned (7, '')
2020-10-05 15:24:27,982 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:24:29 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:24:29,815 - call returned (255, '20/10/05 15:24:29 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:24:29,815 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpKe13ym 2>/tmp/tmppiTNet''] {'quiet': False}
2020-10-05 15:24:29,877 - call returned (7, '')
2020-10-05 15:24:29,878 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:24:31 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:24:31,674 - call returned (255, '20/10/05 15:24:31 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:24:31,675 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:24:31,675 - Will retry 1 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:24:36,680 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpD0lt0R 2>/tmp/tmpsVK0R8''] {'quiet': False}
2020-10-05 15:24:36,741 - call returned (7, '')
2020-10-05 15:24:36,741 - Getting jmx metrics from NN failed. URL: http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmpD0lt0R 2>/tmp/tmpsVK0R8' returned 7. 

2020-10-05 15:24:36,742 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:24:38 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:24:38,545 - call returned (255, '20/10/05 15:24:38 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:24:38,546 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpVrs0nJ 2>/tmp/tmpU_cbgM''] {'quiet': False}
2020-10-05 15:24:38,608 - call returned (7, '')
2020-10-05 15:24:38,609 - Getting jmx metrics from NN failed. URL: http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmpVrs0nJ 2>/tmp/tmpU_cbgM' returned 7. 

2020-10-05 15:24:38,609 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:24:40 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:24:40,449 - call returned (255, '20/10/05 15:24:40 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:24:40,450 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:24:40,450 - No active NameNode was found after 5 retries. Will return current NameNode HA states
2020-10-05 15:24:40,450 - Will retry 107 time(s), caught exception: The NameNode nn1 is not listed as Active or Standby, waiting.... Sleeping for 5 sec(s)
2020-10-05 15:24:45,456 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpnS6y_4 2>/tmp/tmp3WdWsX''] {'quiet': False}
2020-10-05 15:24:45,516 - call returned (7, '')
2020-10-05 15:24:45,516 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:24:47 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:24:47,312 - call returned (255, '20/10/05 15:24:47 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:24:47,313 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpMfkyMW 2>/tmp/tmpseUsf7''] {'quiet': False}
2020-10-05 15:24:47,373 - call returned (7, '')
2020-10-05 15:24:47,374 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:24:49 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:24:49,170 - call returned (255, '20/10/05 15:24:49 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:24:49,171 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:24:49,171 - Will retry 4 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:24:54,177 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpcznvkb 2>/tmp/tmpnElZVu''] {'quiet': False}
2020-10-05 15:24:54,238 - call returned (7, '')
2020-10-05 15:24:54,239 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:24:56 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:24:56,055 - call returned (255, '20/10/05 15:24:56 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:24:56,055 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp7etYmk 2>/tmp/tmphlgH_X''] {'quiet': False}
2020-10-05 15:24:56,115 - call returned (7, '')
2020-10-05 15:24:56,116 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:24:57 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:24:57,915 - call returned (255, '20/10/05 15:24:57 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:24:57,915 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:24:57,915 - Will retry 3 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:25:02,921 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpdstgQF 2>/tmp/tmpomnnuK''] {'quiet': False}
2020-10-05 15:25:02,980 - call returned (7, '')
2020-10-05 15:25:02,981 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:25:04 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:25:04,782 - call returned (255, '20/10/05 15:25:04 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:25:04,782 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpkWRFny 2>/tmp/tmpgqP7cM''] {'quiet': False}
2020-10-05 15:25:04,842 - call returned (7, '')
2020-10-05 15:25:04,843 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:25:06 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:25:06,647 - call returned (255, '20/10/05 15:25:06 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:25:06,647 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:25:06,648 - Will retry 2 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:25:11,653 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpAr4umR 2>/tmp/tmpZhiB9X''] {'quiet': False}
2020-10-05 15:25:11,713 - call returned (7, '')
2020-10-05 15:25:11,713 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:25:13 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:25:13,513 - call returned (255, '20/10/05 15:25:13 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:25:13,514 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmptaHAxF 2>/tmp/tmp8sbP0m''] {'quiet': False}
2020-10-05 15:25:13,575 - call returned (7, '')
2020-10-05 15:25:13,576 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:25:15 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:25:15,393 - call returned (255, '20/10/05 15:25:15 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:25:15,393 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:25:15,394 - Will retry 1 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:25:20,394 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp7p5Ivp 2>/tmp/tmpLzbSYC''] {'quiet': False}
2020-10-05 15:25:20,454 - call returned (7, '')
2020-10-05 15:25:20,455 - Getting jmx metrics from NN failed. URL: http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmp7p5Ivp 2>/tmp/tmpLzbSYC' returned 7. 

2020-10-05 15:25:20,455 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:25:22 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:25:22,250 - call returned (255, '20/10/05 15:25:22 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:25:22,251 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpP5Wu9O 2>/tmp/tmpmUiZSv''] {'quiet': False}
2020-10-05 15:25:22,311 - call returned (7, '')
2020-10-05 15:25:22,311 - Getting jmx metrics from NN failed. URL: http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmpP5Wu9O 2>/tmp/tmpmUiZSv' returned 7. 

2020-10-05 15:25:22,312 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:25:24 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:25:24,110 - call returned (255, '20/10/05 15:25:24 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:25:24,110 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:25:24,111 - No active NameNode was found after 5 retries. Will return current NameNode HA states
2020-10-05 15:25:24,111 - Will retry 106 time(s), caught exception: The NameNode nn1 is not listed as Active or Standby, waiting.... Sleeping for 5 sec(s)
2020-10-05 15:25:29,117 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpW4pP9d 2>/tmp/tmpaRxaxL''] {'quiet': False}
2020-10-05 15:25:29,176 - call returned (7, '')
2020-10-05 15:25:29,177 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:25:30 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:25:30,980 - call returned (255, '20/10/05 15:25:30 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:25:30,980 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpQUikgL 2>/tmp/tmp0SrUwK''] {'quiet': False}
2020-10-05 15:25:31,040 - call returned (7, '')
2020-10-05 15:25:31,041 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:25:32 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:25:32,872 - call returned (255, '20/10/05 15:25:32 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:25:32,872 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:25:32,872 - Will retry 4 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:25:37,878 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpjJPJUM 2>/tmp/tmpfyebFr''] {'quiet': False}
2020-10-05 15:25:37,938 - call returned (7, '')
2020-10-05 15:25:37,939 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:25:39 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:25:39,742 - call returned (255, '20/10/05 15:25:39 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:25:39,743 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpWaGey0 2>/tmp/tmpR_Ksxh''] {'quiet': False}
2020-10-05 15:25:39,807 - call returned (7, '')
2020-10-05 15:25:39,808 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:25:41 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:25:41,690 - call returned (255, '20/10/05 15:25:41 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:25:41,691 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:25:41,691 - Will retry 3 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:25:46,696 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpFqrA9i 2>/tmp/tmpGq4l_p''] {'quiet': False}
2020-10-05 15:25:46,756 - call returned (7, '')
2020-10-05 15:25:46,757 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:25:48 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:25:48,611 - call returned (255, '20/10/05 15:25:48 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:25:48,611 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpZ4JMNw 2>/tmp/tmppVCtxZ''] {'quiet': False}
2020-10-05 15:25:48,671 - call returned (7, '')
2020-10-05 15:25:48,671 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:25:50 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:25:50,471 - call returned (255, '20/10/05 15:25:50 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:25:50,471 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:25:50,471 - Will retry 2 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:25:55,476 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpvxvywG 2>/tmp/tmpOziYvn''] {'quiet': False}
2020-10-05 15:25:55,536 - call returned (7, '')
2020-10-05 15:25:55,536 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:25:57 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:25:57,345 - call returned (255, '20/10/05 15:25:57 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:25:57,346 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmplp2Mts 2>/tmp/tmp72QGDZ''] {'quiet': False}
2020-10-05 15:25:57,406 - call returned (7, '')
2020-10-05 15:25:57,406 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:25:59 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:25:59,203 - call returned (255, '20/10/05 15:25:59 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:25:59,204 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:25:59,204 - Will retry 1 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:26:04,209 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpZ60PyB 2>/tmp/tmpVcpk1D''] {'quiet': False}
2020-10-05 15:26:04,270 - call returned (7, '')
2020-10-05 15:26:04,271 - Getting jmx metrics from NN failed. URL: http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmpZ60PyB 2>/tmp/tmpVcpk1D' returned 7. 

2020-10-05 15:26:04,271 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:26:06 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:26:06,148 - call returned (255, '20/10/05 15:26:06 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:26:06,149 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp6b16P8 2>/tmp/tmp2zIKOU''] {'quiet': False}
2020-10-05 15:26:06,210 - call returned (7, '')
2020-10-05 15:26:06,210 - Getting jmx metrics from NN failed. URL: http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmp6b16P8 2>/tmp/tmp2zIKOU' returned 7. 

2020-10-05 15:26:06,211 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:26:08 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:26:08,060 - call returned (255, '20/10/05 15:26:08 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:26:08,061 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:26:08,061 - No active NameNode was found after 5 retries. Will return current NameNode HA states
2020-10-05 15:26:08,061 - Will retry 105 time(s), caught exception: The NameNode nn1 is not listed as Active or Standby, waiting.... Sleeping for 5 sec(s)
2020-10-05 15:26:13,067 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpJeja8P 2>/tmp/tmpiRgZgb''] {'quiet': False}
2020-10-05 15:26:13,126 - call returned (7, '')
2020-10-05 15:26:13,127 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:26:14 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:26:14,930 - call returned (255, '20/10/05 15:26:14 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:26:14,931 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpuiHGf3 2>/tmp/tmpyerMZD''] {'quiet': False}
2020-10-05 15:26:14,991 - call returned (7, '')
2020-10-05 15:26:14,991 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:26:16 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:26:16,787 - call returned (255, '20/10/05 15:26:16 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:26:16,788 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:26:16,788 - Will retry 4 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:26:21,793 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmplbu1nF 2>/tmp/tmpYGFe8A''] {'quiet': False}
2020-10-05 15:26:21,854 - call returned (7, '')
2020-10-05 15:26:21,855 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:26:23 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:26:23,648 - call returned (255, '20/10/05 15:26:23 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:26:23,649 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpKtp8d0 2>/tmp/tmp3oqJ6v''] {'quiet': False}
2020-10-05 15:26:23,709 - call returned (7, '')
2020-10-05 15:26:23,710 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:26:25 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:26:25,552 - call returned (255, '20/10/05 15:26:25 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:26:25,553 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:26:25,553 - Will retry 3 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:26:30,556 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpz_brAb 2>/tmp/tmpAjDJEo''] {'quiet': False}
2020-10-05 15:26:30,619 - call returned (7, '')
2020-10-05 15:26:30,620 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:26:32 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:26:32,439 - call returned (255, '20/10/05 15:26:32 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:26:32,440 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpxQABwo 2>/tmp/tmpLn4iP5''] {'quiet': False}
2020-10-05 15:26:32,504 - call returned (7, '')
2020-10-05 15:26:32,504 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:26:34 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:26:34,366 - call returned (255, '20/10/05 15:26:34 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:26:34,366 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:26:34,367 - Will retry 2 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:26:39,372 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpe1EjGk 2>/tmp/tmpldB_0i''] {'quiet': False}
2020-10-05 15:26:39,432 - call returned (7, '')
2020-10-05 15:26:39,432 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:26:41 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:26:41,235 - call returned (255, '20/10/05 15:26:41 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:26:41,236 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpqUPSHw 2>/tmp/tmpQ6nQWL''] {'quiet': False}
2020-10-05 15:26:41,296 - call returned (7, '')
2020-10-05 15:26:41,297 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:26:43 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:26:43,094 - call returned (255, '20/10/05 15:26:43 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:26:43,094 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:26:43,095 - Will retry 1 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:26:48,100 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpVkQwmz 2>/tmp/tmpUoLeAR''] {'quiet': False}
2020-10-05 15:26:48,161 - call returned (7, '')
2020-10-05 15:26:48,161 - Getting jmx metrics from NN failed. URL: http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmpVkQwmz 2>/tmp/tmpUoLeAR' returned 7. 

2020-10-05 15:26:48,162 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:26:49 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:26:49,974 - call returned (255, '20/10/05 15:26:49 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:26:49,975 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpGoLb6B 2>/tmp/tmpWJejJR''] {'quiet': False}
2020-10-05 15:26:50,035 - call returned (7, '')
2020-10-05 15:26:50,036 - Getting jmx metrics from NN failed. URL: http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmpGoLb6B 2>/tmp/tmpWJejJR' returned 7. 

2020-10-05 15:26:50,036 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:26:51 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:26:51,827 - call returned (255, '20/10/05 15:26:51 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:26:51,827 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:26:51,827 - No active NameNode was found after 5 retries. Will return current NameNode HA states
2020-10-05 15:26:51,828 - Will retry 104 time(s), caught exception: The NameNode nn1 is not listed as Active or Standby, waiting.... Sleeping for 5 sec(s)
2020-10-05 15:26:56,833 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpCNj72S 2>/tmp/tmpp3LTr5''] {'quiet': False}
2020-10-05 15:26:56,894 - call returned (7, '')
2020-10-05 15:26:56,895 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:26:58 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:26:58,755 - call returned (255, '20/10/05 15:26:58 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:26:58,755 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpPrP8Jz 2>/tmp/tmpY3J4JY''] {'quiet': False}
2020-10-05 15:26:58,820 - call returned (7, '')
2020-10-05 15:26:58,821 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:27:00 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:27:00,648 - call returned (255, '20/10/05 15:27:00 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:27:00,648 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:27:00,648 - Will retry 4 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:27:05,654 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp6muZNC 2>/tmp/tmpjDmbEB''] {'quiet': False}
2020-10-05 15:27:05,715 - call returned (7, '')
2020-10-05 15:27:05,715 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:27:07 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:27:07,523 - call returned (255, '20/10/05 15:27:07 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:27:07,524 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmprSleau 2>/tmp/tmpYiLwwn''] {'quiet': False}
2020-10-05 15:27:07,584 - call returned (7, '')
2020-10-05 15:27:07,584 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:27:09 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:27:09,412 - call returned (255, '20/10/05 15:27:09 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:27:09,412 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:27:09,412 - Will retry 3 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:27:14,415 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpKmSLU4 2>/tmp/tmpbWPerR''] {'quiet': False}
2020-10-05 15:27:14,474 - call returned (7, '')
2020-10-05 15:27:14,474 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:27:16 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:27:16,330 - call returned (255, '20/10/05 15:27:16 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:27:16,331 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpG1cB_k 2>/tmp/tmpAM_W1L''] {'quiet': False}
2020-10-05 15:27:16,391 - call returned (7, '')
2020-10-05 15:27:16,391 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:27:18 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:27:18,188 - call returned (255, '20/10/05 15:27:18 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:27:18,189 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:27:18,189 - Will retry 2 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:27:23,194 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpfDSU4c 2>/tmp/tmpm3TktI''] {'quiet': False}
2020-10-05 15:27:23,257 - call returned (7, '')
2020-10-05 15:27:23,257 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:27:25 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:27:25,087 - call returned (255, '20/10/05 15:27:25 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:27:25,087 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpMzRIv0 2>/tmp/tmp81Z4Ok''] {'quiet': False}
2020-10-05 15:27:25,147 - call returned (7, '')
2020-10-05 15:27:25,147 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:27:26 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:27:26,941 - call returned (255, '20/10/05 15:27:26 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:27:26,942 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:27:26,942 - Will retry 1 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:27:31,948 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpnwstOx 2>/tmp/tmpI7jr4s''] {'quiet': False}
2020-10-05 15:27:32,008 - call returned (7, '')
2020-10-05 15:27:32,008 - Getting jmx metrics from NN failed. URL: http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmpnwstOx 2>/tmp/tmpI7jr4s' returned 7. 

2020-10-05 15:27:32,009 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:27:33 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:27:33,808 - call returned (255, '20/10/05 15:27:33 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:27:33,808 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp1fAM3I 2>/tmp/tmpvSqFXJ''] {'quiet': False}
2020-10-05 15:27:33,868 - call returned (7, '')
2020-10-05 15:27:33,869 - Getting jmx metrics from NN failed. URL: http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmp1fAM3I 2>/tmp/tmpvSqFXJ' returned 7. 

2020-10-05 15:27:33,869 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:27:35 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:27:35,665 - call returned (255, '20/10/05 15:27:35 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:27:35,665 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:27:35,665 - No active NameNode was found after 5 retries. Will return current NameNode HA states
2020-10-05 15:27:35,665 - Will retry 103 time(s), caught exception: The NameNode nn1 is not listed as Active or Standby, waiting.... Sleeping for 5 sec(s)
2020-10-05 15:27:40,671 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpfFKEg2 2>/tmp/tmpsA3MG7''] {'quiet': False}
2020-10-05 15:27:40,731 - call returned (7, '')
2020-10-05 15:27:40,731 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:27:42 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:27:42,567 - call returned (255, '20/10/05 15:27:42 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:27:42,567 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpIR4_wn 2>/tmp/tmpXokqHb''] {'quiet': False}
2020-10-05 15:27:42,628 - call returned (7, '')
2020-10-05 15:27:42,629 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:27:44 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:27:44,430 - call returned (255, '20/10/05 15:27:44 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:27:44,430 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:27:44,430 - Will retry 4 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:27:49,436 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpILWNNU 2>/tmp/tmpG_B2HL''] {'quiet': False}
2020-10-05 15:27:49,496 - call returned (7, '')
2020-10-05 15:27:49,497 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:27:51 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:27:51,298 - call returned (255, '20/10/05 15:27:51 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:27:51,299 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpp7w30K 2>/tmp/tmph0sdPJ''] {'quiet': False}
2020-10-05 15:27:51,359 - call returned (7, '')
2020-10-05 15:27:51,359 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:27:53 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:27:53,161 - call returned (255, '20/10/05 15:27:53 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:27:53,161 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:27:53,161 - Will retry 3 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:27:58,167 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpZ2jNou 2>/tmp/tmp0gWYWx''] {'quiet': False}
2020-10-05 15:27:58,228 - call returned (7, '')
2020-10-05 15:27:58,228 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:28:00 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:28:00,027 - call returned (255, '20/10/05 15:28:00 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:28:00,028 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpqemtKY 2>/tmp/tmphd6Typ''] {'quiet': False}
2020-10-05 15:28:00,089 - call returned (7, '')
2020-10-05 15:28:00,089 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:28:01 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:28:01,896 - call returned (255, '20/10/05 15:28:01 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:28:01,896 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:28:01,896 - Will retry 2 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:28:06,902 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpl6qoeV 2>/tmp/tmpZCnuZ7''] {'quiet': False}
2020-10-05 15:28:06,962 - call returned (7, '')
2020-10-05 15:28:06,962 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:28:08 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:28:08,755 - call returned (255, '20/10/05 15:28:08 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:28:08,755 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpDEv5xT 2>/tmp/tmpLmcVJe''] {'quiet': False}
2020-10-05 15:28:08,815 - call returned (7, '')
2020-10-05 15:28:08,816 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:28:10 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:28:10,623 - call returned (255, '20/10/05 15:28:10 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:28:10,623 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:28:10,623 - Will retry 1 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:28:15,629 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpHnQ6Xu 2>/tmp/tmpZ8E5Nw''] {'quiet': False}
2020-10-05 15:28:15,691 - call returned (7, '')
2020-10-05 15:28:15,692 - Getting jmx metrics from NN failed. URL: http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmpHnQ6Xu 2>/tmp/tmpZ8E5Nw' returned 7. 

2020-10-05 15:28:15,692 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:28:17 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:28:17,506 - call returned (255, '20/10/05 15:28:17 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:28:17,507 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpAm7ZVF 2>/tmp/tmpxvobTl''] {'quiet': False}
2020-10-05 15:28:17,566 - call returned (7, '')
2020-10-05 15:28:17,566 - Getting jmx metrics from NN failed. URL: http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmpAm7ZVF 2>/tmp/tmpxvobTl' returned 7. 

2020-10-05 15:28:17,567 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:28:19 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:28:19,360 - call returned (255, '20/10/05 15:28:19 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:28:19,360 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:28:19,360 - No active NameNode was found after 5 retries. Will return current NameNode HA states
2020-10-05 15:28:19,361 - Will retry 102 time(s), caught exception: The NameNode nn1 is not listed as Active or Standby, waiting.... Sleeping for 5 sec(s)
2020-10-05 15:28:24,367 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpLcIlhB 2>/tmp/tmpPxbw3Q''] {'quiet': False}
2020-10-05 15:28:24,426 - call returned (7, '')
2020-10-05 15:28:24,426 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:28:26 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:28:26,223 - call returned (255, '20/10/05 15:28:26 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:28:26,224 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp_IvUT4 2>/tmp/tmpMvkeGV''] {'quiet': False}
2020-10-05 15:28:26,284 - call returned (7, '')
2020-10-05 15:28:26,285 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:28:28 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:28:28,106 - call returned (255, '20/10/05 15:28:28 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:28:28,106 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:28:28,106 - Will retry 4 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:28:33,112 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpmbaxTv 2>/tmp/tmp8K43RH''] {'quiet': False}
2020-10-05 15:28:33,171 - call returned (7, '')
2020-10-05 15:28:33,172 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:28:34 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:28:34,960 - call returned (255, '20/10/05 15:28:34 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:28:34,961 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp4A0z7_ 2>/tmp/tmpZprGo3''] {'quiet': False}
2020-10-05 15:28:35,022 - call returned (7, '')
2020-10-05 15:28:35,023 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:28:36 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:28:36,918 - call returned (255, '20/10/05 15:28:36 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:28:36,919 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:28:36,919 - Will retry 3 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:28:41,922 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpEe1mXf 2>/tmp/tmpQYIrvW''] {'quiet': False}
2020-10-05 15:28:41,993 - call returned (7, '')
2020-10-05 15:28:41,994 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:28:43 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:28:43,918 - call returned (255, '20/10/05 15:28:43 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:28:43,919 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp94MqHQ 2>/tmp/tmpChLCQk''] {'quiet': False}
2020-10-05 15:28:43,979 - call returned (7, '')
2020-10-05 15:28:43,979 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:28:45 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:28:45,777 - call returned (255, '20/10/05 15:28:45 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:28:45,777 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:28:45,777 - Will retry 2 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:28:50,783 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpLdydv2 2>/tmp/tmpZmvKbc''] {'quiet': False}
2020-10-05 15:28:50,843 - call returned (7, '')
2020-10-05 15:28:50,843 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:28:52 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:28:52,642 - call returned (255, '20/10/05 15:28:52 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:28:52,642 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp60RKLD 2>/tmp/tmpeI6g7w''] {'quiet': False}
2020-10-05 15:28:52,701 - call returned (7, '')
2020-10-05 15:28:52,702 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:28:54 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:28:54,500 - call returned (255, '20/10/05 15:28:54 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:28:54,500 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:28:54,500 - Will retry 1 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:28:59,506 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpzqW3QL 2>/tmp/tmpe8J22s''] {'quiet': False}
2020-10-05 15:28:59,567 - call returned (7, '')
2020-10-05 15:28:59,568 - Getting jmx metrics from NN failed. URL: http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmpzqW3QL 2>/tmp/tmpe8J22s' returned 7. 

2020-10-05 15:28:59,568 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:29:01 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:29:01,400 - call returned (255, '20/10/05 15:29:01 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:29:01,401 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpVh_hkM 2>/tmp/tmpH59vfi''] {'quiet': False}
2020-10-05 15:29:01,462 - call returned (7, '')
2020-10-05 15:29:01,463 - Getting jmx metrics from NN failed. URL: http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmpVh_hkM 2>/tmp/tmpH59vfi' returned 7. 

2020-10-05 15:29:01,463 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:29:03 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:29:03,330 - call returned (255, '20/10/05 15:29:03 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:29:03,330 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:29:03,331 - No active NameNode was found after 5 retries. Will return current NameNode HA states
2020-10-05 15:29:03,331 - Will retry 101 time(s), caught exception: The NameNode nn1 is not listed as Active or Standby, waiting.... Sleeping for 5 sec(s)
2020-10-05 15:29:08,337 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpXRJ7by 2>/tmp/tmpuj33GR''] {'quiet': False}
2020-10-05 15:29:08,397 - call returned (7, '')
2020-10-05 15:29:08,398 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:29:10 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:29:10,188 - call returned (255, '20/10/05 15:29:10 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:29:10,189 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpJcIrB2 2>/tmp/tmpUL2p2A''] {'quiet': False}
2020-10-05 15:29:10,249 - call returned (7, '')
2020-10-05 15:29:10,250 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:29:12 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:29:12,052 - call returned (255, '20/10/05 15:29:12 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:29:12,052 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:29:12,052 - Will retry 4 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:29:17,058 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpFC6ACQ 2>/tmp/tmpal9BGd''] {'quiet': False}
2020-10-05 15:29:17,118 - call returned (7, '')
2020-10-05 15:29:17,118 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:29:18 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:29:18,918 - call returned (255, '20/10/05 15:29:18 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:29:18,919 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpR_iivO 2>/tmp/tmppFdzWX''] {'quiet': False}
2020-10-05 15:29:18,978 - call returned (7, '')
2020-10-05 15:29:18,979 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:29:20 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:29:20,774 - call returned (255, '20/10/05 15:29:20 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:29:20,774 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:29:20,775 - Will retry 3 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:29:25,780 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpLqa_FK 2>/tmp/tmpbVOfvP''] {'quiet': False}
2020-10-05 15:29:25,841 - call returned (7, '')
2020-10-05 15:29:25,842 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:29:27 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:29:27,672 - call returned (255, '20/10/05 15:29:27 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:29:27,672 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpCBU1J5 2>/tmp/tmpgvvAql''] {'quiet': False}
2020-10-05 15:29:27,735 - call returned (7, '')
2020-10-05 15:29:27,736 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:29:29 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:29:29,558 - call returned (255, '20/10/05 15:29:29 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:29:29,559 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:29:29,559 - Will retry 2 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:29:34,564 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpj6jKTn 2>/tmp/tmppp3KWT''] {'quiet': False}
2020-10-05 15:29:34,624 - call returned (7, '')
2020-10-05 15:29:34,624 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:29:36 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:29:36,419 - call returned (255, '20/10/05 15:29:36 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:29:36,420 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpQGn5Ht 2>/tmp/tmpnt8Fob''] {'quiet': False}
2020-10-05 15:29:36,480 - call returned (7, '')
2020-10-05 15:29:36,480 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:29:38 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:29:38,279 - call returned (255, '20/10/05 15:29:38 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:29:38,280 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:29:38,280 - Will retry 1 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:29:43,285 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp58TwrQ 2>/tmp/tmp_AGVKU''] {'quiet': False}
2020-10-05 15:29:43,345 - call returned (7, '')
2020-10-05 15:29:43,345 - Getting jmx metrics from NN failed. URL: http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmp58TwrQ 2>/tmp/tmp_AGVKU' returned 7. 

2020-10-05 15:29:43,346 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:29:45 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:29:45,152 - call returned (255, '20/10/05 15:29:45 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:29:45,152 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpxSwgeh 2>/tmp/tmp3oLPWH''] {'quiet': False}
2020-10-05 15:29:45,212 - call returned (7, '')
2020-10-05 15:29:45,212 - Getting jmx metrics from NN failed. URL: http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmpxSwgeh 2>/tmp/tmp3oLPWH' returned 7. 

2020-10-05 15:29:45,213 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:29:47 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:29:47,016 - call returned (255, '20/10/05 15:29:47 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:29:47,016 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:29:47,017 - No active NameNode was found after 5 retries. Will return current NameNode HA states
2020-10-05 15:29:47,017 - Will retry 100 time(s), caught exception: The NameNode nn1 is not listed as Active or Standby, waiting.... Sleeping for 5 sec(s)
2020-10-05 15:29:52,023 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpuVXU0W 2>/tmp/tmpLrgJf1''] {'quiet': False}
2020-10-05 15:29:52,085 - call returned (7, '')
2020-10-05 15:29:52,085 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:29:53 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:29:53,910 - call returned (255, '20/10/05 15:29:53 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:29:53,910 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpotijXs 2>/tmp/tmpG7su9m''] {'quiet': False}
2020-10-05 15:29:53,971 - call returned (7, '')
2020-10-05 15:29:53,972 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:29:55 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:29:55,788 - call returned (255, '20/10/05 15:29:55 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:29:55,788 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:29:55,789 - Will retry 4 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:30:00,794 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp5baynj 2>/tmp/tmpJqq81u''] {'quiet': False}
2020-10-05 15:30:00,854 - call returned (7, '')
2020-10-05 15:30:00,855 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:30:02 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:30:02,674 - call returned (255, '20/10/05 15:30:02 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:30:02,674 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpc00dPS 2>/tmp/tmp7LRU8Y''] {'quiet': False}
2020-10-05 15:30:02,734 - call returned (7, '')
2020-10-05 15:30:02,735 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:30:04 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:30:04,528 - call returned (255, '20/10/05 15:30:04 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:30:04,528 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:30:04,529 - Will retry 3 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:30:09,534 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpaa2gBz 2>/tmp/tmp7X544H''] {'quiet': False}
2020-10-05 15:30:09,594 - call returned (7, '')
2020-10-05 15:30:09,595 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:30:11 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:30:11,425 - call returned (255, '20/10/05 15:30:11 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:30:11,426 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpoJb1R6 2>/tmp/tmpuhDEgv''] {'quiet': False}
2020-10-05 15:30:11,486 - call returned (7, '')
2020-10-05 15:30:11,487 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:30:13 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:30:13,312 - call returned (255, '20/10/05 15:30:13 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:30:13,312 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:30:13,313 - Will retry 2 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:30:18,318 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpJFWZWs 2>/tmp/tmpurBhjq''] {'quiet': False}
2020-10-05 15:30:18,381 - call returned (7, '')
2020-10-05 15:30:18,381 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:30:20 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:30:20,251 - call returned (255, '20/10/05 15:30:20 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:30:20,251 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp2D4QNI 2>/tmp/tmpN_5zFB''] {'quiet': False}
2020-10-05 15:30:20,310 - call returned (7, '')
2020-10-05 15:30:20,311 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:30:22 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:30:22,106 - call returned (255, '20/10/05 15:30:22 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:30:22,106 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:30:22,107 - Will retry 1 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:30:27,112 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp8QPDix 2>/tmp/tmpbeun8v''] {'quiet': False}
2020-10-05 15:30:27,172 - call returned (7, '')
2020-10-05 15:30:27,172 - Getting jmx metrics from NN failed. URL: http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmp8QPDix 2>/tmp/tmpbeun8v' returned 7. 

2020-10-05 15:30:27,173 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:30:28 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:30:28,977 - call returned (255, '20/10/05 15:30:28 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:30:28,978 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpgiDZV6 2>/tmp/tmpLUXuKc''] {'quiet': False}
2020-10-05 15:30:29,038 - call returned (7, '')
2020-10-05 15:30:29,039 - Getting jmx metrics from NN failed. URL: http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmpgiDZV6 2>/tmp/tmpLUXuKc' returned 7. 

2020-10-05 15:30:29,039 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:30:30 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:30:30,840 - call returned (255, '20/10/05 15:30:30 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:30:30,840 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:30:30,840 - No active NameNode was found after 5 retries. Will return current NameNode HA states
2020-10-05 15:30:30,841 - Will retry 99 time(s), caught exception: The NameNode nn1 is not listed as Active or Standby, waiting.... Sleeping for 5 sec(s)
2020-10-05 15:30:35,846 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpn7VTjC 2>/tmp/tmpKR6gAR''] {'quiet': False}
2020-10-05 15:30:35,908 - call returned (7, '')
2020-10-05 15:30:35,909 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:30:37 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:30:37,779 - call returned (255, '20/10/05 15:30:37 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:30:37,779 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpGmCI9x 2>/tmp/tmp2MELju''] {'quiet': False}
2020-10-05 15:30:37,839 - call returned (7, '')
2020-10-05 15:30:37,840 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:30:39 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:30:39,651 - call returned (255, '20/10/05 15:30:39 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:30:39,651 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:30:39,652 - Will retry 4 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:30:44,657 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpKFkTOb 2>/tmp/tmpRALlf2''] {'quiet': False}
2020-10-05 15:30:44,717 - call returned (7, '')
2020-10-05 15:30:44,718 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:30:46 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:30:46,546 - call returned (255, '20/10/05 15:30:46 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:30:46,547 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpidBP6b 2>/tmp/tmpA2cLfa''] {'quiet': False}
2020-10-05 15:30:46,607 - call returned (7, '')
2020-10-05 15:30:46,608 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:30:48 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:30:48,401 - call returned (255, '20/10/05 15:30:48 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:30:48,401 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:30:48,401 - Will retry 3 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:30:53,402 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpQDx1Yw 2>/tmp/tmphTBCjL''] {'quiet': False}
2020-10-05 15:30:53,463 - call returned (7, '')
2020-10-05 15:30:53,463 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:30:55 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:30:55,263 - call returned (255, '20/10/05 15:30:55 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:30:55,264 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpARRZXP 2>/tmp/tmpfDOfFM''] {'quiet': False}
2020-10-05 15:30:55,324 - call returned (7, '')
2020-10-05 15:30:55,324 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:30:57 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:30:57,117 - call returned (255, '20/10/05 15:30:57 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:30:57,118 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:30:57,118 - Will retry 2 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:31:02,123 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpNYXUyP 2>/tmp/tmpYkDU3S''] {'quiet': False}
2020-10-05 15:31:02,183 - call returned (7, '')
2020-10-05 15:31:02,184 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:31:04 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:31:04,051 - call returned (255, '20/10/05 15:31:04 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:31:04,052 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpzRCKZj 2>/tmp/tmpA7QKlF''] {'quiet': False}
2020-10-05 15:31:04,111 - call returned (7, '')
2020-10-05 15:31:04,112 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:31:05 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:31:05,951 - call returned (255, '20/10/05 15:31:05 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:31:05,951 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:31:05,952 - Will retry 1 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:31:10,957 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp5dzClR 2>/tmp/tmpMNFrCI''] {'quiet': False}
2020-10-05 15:31:11,017 - call returned (7, '')
2020-10-05 15:31:11,017 - Getting jmx metrics from NN failed. URL: http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmp5dzClR 2>/tmp/tmpMNFrCI' returned 7. 

2020-10-05 15:31:11,018 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:31:12 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:31:12,816 - call returned (255, '20/10/05 15:31:12 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:31:12,817 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmppfQklV 2>/tmp/tmplA9M6M''] {'quiet': False}
2020-10-05 15:31:12,876 - call returned (7, '')
2020-10-05 15:31:12,877 - Getting jmx metrics from NN failed. URL: http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmppfQklV 2>/tmp/tmplA9M6M' returned 7. 

2020-10-05 15:31:12,877 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:31:14 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:31:14,678 - call returned (255, '20/10/05 15:31:14 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:31:14,678 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:31:14,679 - No active NameNode was found after 5 retries. Will return current NameNode HA states
2020-10-05 15:31:14,679 - Will retry 98 time(s), caught exception: The NameNode nn1 is not listed as Active or Standby, waiting.... Sleeping for 5 sec(s)
2020-10-05 15:31:19,685 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmplGjy61 2>/tmp/tmpJEynuu''] {'quiet': False}
2020-10-05 15:31:19,743 - call returned (7, '')
2020-10-05 15:31:19,744 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:31:21 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:31:21,540 - call returned (255, '20/10/05 15:31:21 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:31:21,541 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpkslhJK 2>/tmp/tmpM4V7lO''] {'quiet': False}
2020-10-05 15:31:21,601 - call returned (7, '')
2020-10-05 15:31:21,602 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:31:23 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:31:23,408 - call returned (255, '20/10/05 15:31:23 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:31:23,408 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:31:23,409 - Will retry 4 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:31:28,414 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpjsmjEQ 2>/tmp/tmpkMDyVP''] {'quiet': False}
2020-10-05 15:31:28,474 - call returned (7, '')
2020-10-05 15:31:28,475 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:31:30 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:31:30,271 - call returned (255, '20/10/05 15:31:30 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:31:30,272 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpiaZ03Q 2>/tmp/tmpPAGIPy''] {'quiet': False}
2020-10-05 15:31:30,333 - call returned (7, '')
2020-10-05 15:31:30,334 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:31:32 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:31:32,160 - call returned (255, '20/10/05 15:31:32 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:31:32,160 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:31:32,160 - Will retry 3 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:31:37,166 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpJeTSMN 2>/tmp/tmpefEeoR''] {'quiet': False}
2020-10-05 15:31:37,228 - call returned (7, '')
2020-10-05 15:31:37,229 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:31:39 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:31:39,125 - call returned (255, '20/10/05 15:31:39 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:31:39,126 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpPbiznk 2>/tmp/tmp5VTvCP''] {'quiet': False}
2020-10-05 15:31:39,187 - call returned (7, '')
2020-10-05 15:31:39,187 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:31:41 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:31:41,042 - call returned (255, '20/10/05 15:31:41 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:31:41,043 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:31:41,043 - Will retry 2 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:31:46,048 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpEdFbtu 2>/tmp/tmpDvhKfi''] {'quiet': False}
2020-10-05 15:31:46,109 - call returned (7, '')
2020-10-05 15:31:46,109 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:31:47 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:31:47,959 - call returned (255, '20/10/05 15:31:47 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:31:47,960 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp8Dz5W8 2>/tmp/tmpoUP29T''] {'quiet': False}
2020-10-05 15:31:48,021 - call returned (7, '')
2020-10-05 15:31:48,022 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:31:49 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:31:49,817 - call returned (255, '20/10/05 15:31:49 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:31:49,818 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:31:49,818 - Will retry 1 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:31:54,823 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpqE71yC 2>/tmp/tmp0isJ6H''] {'quiet': False}
2020-10-05 15:31:54,885 - call returned (7, '')
2020-10-05 15:31:54,886 - Getting jmx metrics from NN failed. URL: http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmpqE71yC 2>/tmp/tmp0isJ6H' returned 7. 

2020-10-05 15:31:54,886 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:31:56 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:31:56,714 - call returned (255, '20/10/05 15:31:56 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:31:56,714 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpnQVzAe 2>/tmp/tmpihs8YK''] {'quiet': False}
2020-10-05 15:31:56,775 - call returned (7, '')
2020-10-05 15:31:56,776 - Getting jmx metrics from NN failed. URL: http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmpnQVzAe 2>/tmp/tmpihs8YK' returned 7. 

2020-10-05 15:31:56,776 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:31:58 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:31:58,604 - call returned (255, '20/10/05 15:31:58 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:31:58,604 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:31:58,604 - No active NameNode was found after 5 retries. Will return current NameNode HA states
2020-10-05 15:31:58,604 - Will retry 97 time(s), caught exception: The NameNode nn1 is not listed as Active or Standby, waiting.... Sleeping for 5 sec(s)
2020-10-05 15:32:03,610 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpuqFczk 2>/tmp/tmpVSqPjg''] {'quiet': False}
2020-10-05 15:32:03,671 - call returned (7, '')
2020-10-05 15:32:03,671 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:32:05 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:32:05,467 - call returned (255, '20/10/05 15:32:05 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:32:05,468 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpEbS1aJ 2>/tmp/tmp4YANxZ''] {'quiet': False}
2020-10-05 15:32:05,528 - call returned (7, '')
2020-10-05 15:32:05,529 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:32:07 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:32:07,330 - call returned (255, '20/10/05 15:32:07 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:32:07,330 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:32:07,330 - Will retry 4 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:32:12,336 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp5ZPjnZ 2>/tmp/tmpVMIDFG''] {'quiet': False}
2020-10-05 15:32:12,396 - call returned (7, '')
2020-10-05 15:32:12,397 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:32:14 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:32:14,187 - call returned (255, '20/10/05 15:32:14 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:32:14,188 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpMvwFup 2>/tmp/tmp3qfci_''] {'quiet': False}
2020-10-05 15:32:14,248 - call returned (7, '')
2020-10-05 15:32:14,249 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:32:16 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:32:16,043 - call returned (255, '20/10/05 15:32:16 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:32:16,044 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:32:16,044 - Will retry 3 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:32:21,049 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpNJ108b 2>/tmp/tmpuAqGXI''] {'quiet': False}
2020-10-05 15:32:21,111 - call returned (7, '')
2020-10-05 15:32:21,112 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:32:22 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:32:22,956 - call returned (255, '20/10/05 15:32:22 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:32:22,957 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpYEMrmE 2>/tmp/tmpaEfFEZ''] {'quiet': False}
2020-10-05 15:32:23,018 - call returned (7, '')
2020-10-05 15:32:23,019 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:32:24 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:32:24,838 - call returned (255, '20/10/05 15:32:24 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:32:24,838 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:32:24,839 - Will retry 2 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:32:29,844 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpPLSBmz 2>/tmp/tmpQfOKwP''] {'quiet': False}
2020-10-05 15:32:29,904 - call returned (7, '')
2020-10-05 15:32:29,905 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:32:31 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:32:31,723 - call returned (255, '20/10/05 15:32:31 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:32:31,724 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpDr0vrz 2>/tmp/tmphTUcst''] {'quiet': False}
2020-10-05 15:32:31,784 - call returned (7, '')
2020-10-05 15:32:31,784 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:32:33 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:32:33,582 - call returned (255, '20/10/05 15:32:33 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:32:33,582 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:32:33,582 - Will retry 1 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:32:38,588 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmptDfxmA 2>/tmp/tmpQd5g4I''] {'quiet': False}
2020-10-05 15:32:38,647 - call returned (7, '')
2020-10-05 15:32:38,647 - Getting jmx metrics from NN failed. URL: http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmptDfxmA 2>/tmp/tmpQd5g4I' returned 7. 

2020-10-05 15:32:38,648 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:32:40 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:32:40,491 - call returned (255, '20/10/05 15:32:40 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:32:40,492 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpNMyAFY 2>/tmp/tmp5CTUiv''] {'quiet': False}
2020-10-05 15:32:40,556 - call returned (7, '')
2020-10-05 15:32:40,557 - Getting jmx metrics from NN failed. URL: http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmpNMyAFY 2>/tmp/tmp5CTUiv' returned 7. 

2020-10-05 15:32:40,557 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:32:42 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:32:42,421 - call returned (255, '20/10/05 15:32:42 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:32:42,421 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:32:42,422 - No active NameNode was found after 5 retries. Will return current NameNode HA states
2020-10-05 15:32:42,422 - Will retry 96 time(s), caught exception: The NameNode nn1 is not listed as Active or Standby, waiting.... Sleeping for 5 sec(s)
2020-10-05 15:32:47,428 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpk2igJ2 2>/tmp/tmpdrDUwM''] {'quiet': False}
2020-10-05 15:32:47,489 - call returned (7, '')
2020-10-05 15:32:47,489 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:32:49 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:32:49,317 - call returned (255, '20/10/05 15:32:49 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:32:49,318 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpfKOmKa 2>/tmp/tmpPXiVro''] {'quiet': False}
2020-10-05 15:32:49,379 - call returned (7, '')
2020-10-05 15:32:49,380 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:32:51 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:32:51,190 - call returned (255, '20/10/05 15:32:51 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:32:51,190 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:32:51,190 - Will retry 4 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:32:56,196 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpUKr8X0 2>/tmp/tmpFvgBP9''] {'quiet': False}
2020-10-05 15:32:56,256 - call returned (7, '')
2020-10-05 15:32:56,257 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:32:58 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:32:58,050 - call returned (255, '20/10/05 15:32:58 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:32:58,051 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpG4MT_R 2>/tmp/tmpOu7tbP''] {'quiet': False}
2020-10-05 15:32:58,114 - call returned (7, '')
2020-10-05 15:32:58,114 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:32:59 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:32:59,904 - call returned (255, '20/10/05 15:32:59 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:32:59,904 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:32:59,904 - Will retry 3 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:33:04,910 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp4Hjnrq 2>/tmp/tmpDIq7bG''] {'quiet': False}
2020-10-05 15:33:04,969 - call returned (7, '')
2020-10-05 15:33:04,969 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:33:06 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:33:06,763 - call returned (255, '20/10/05 15:33:06 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:33:06,764 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp4arCW3 2>/tmp/tmpJ3IxUL''] {'quiet': False}
2020-10-05 15:33:06,823 - call returned (7, '')
2020-10-05 15:33:06,824 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:33:08 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:33:08,645 - call returned (255, '20/10/05 15:33:08 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:33:08,645 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:33:08,646 - Will retry 2 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:33:13,651 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpvpG2iQ 2>/tmp/tmpZjWygx''] {'quiet': False}
2020-10-05 15:33:13,712 - call returned (7, '')
2020-10-05 15:33:13,713 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:33:15 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:33:15,541 - call returned (255, '20/10/05 15:33:15 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:33:15,541 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpo2ROLT 2>/tmp/tmpf9hS5s''] {'quiet': False}
2020-10-05 15:33:15,601 - call returned (7, '')
2020-10-05 15:33:15,601 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:33:17 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:33:17,398 - call returned (255, '20/10/05 15:33:17 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:33:17,398 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:33:17,398 - Will retry 1 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:33:22,404 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp1VnK19 2>/tmp/tmpTiXaA3''] {'quiet': False}
2020-10-05 15:33:22,464 - call returned (7, '')
2020-10-05 15:33:22,465 - Getting jmx metrics from NN failed. URL: http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmp1VnK19 2>/tmp/tmpTiXaA3' returned 7. 

2020-10-05 15:33:22,465 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:33:24 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:33:24,273 - call returned (255, '20/10/05 15:33:24 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:33:24,273 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmplq_UNp 2>/tmp/tmpJWApWk''] {'quiet': False}
2020-10-05 15:33:24,334 - call returned (7, '')
2020-10-05 15:33:24,335 - Getting jmx metrics from NN failed. URL: http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmplq_UNp 2>/tmp/tmpJWApWk' returned 7. 

2020-10-05 15:33:24,335 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:33:26 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:33:26,138 - call returned (255, '20/10/05 15:33:26 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:33:26,139 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:33:26,139 - No active NameNode was found after 5 retries. Will return current NameNode HA states
2020-10-05 15:33:26,139 - Will retry 95 time(s), caught exception: The NameNode nn1 is not listed as Active or Standby, waiting.... Sleeping for 5 sec(s)
2020-10-05 15:33:31,145 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpVC7psM 2>/tmp/tmpgGc8kb''] {'quiet': False}
2020-10-05 15:33:31,204 - call returned (7, '')
2020-10-05 15:33:31,205 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:33:32 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:33:33,001 - call returned (255, '20/10/05 15:33:32 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:33:33,002 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmptAAN_7 2>/tmp/tmp0MZiyv''] {'quiet': False}
2020-10-05 15:33:33,063 - call returned (7, '')
2020-10-05 15:33:33,064 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:33:34 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:33:34,864 - call returned (255, '20/10/05 15:33:34 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:33:34,864 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:33:34,864 - Will retry 4 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:33:39,870 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpqadreS 2>/tmp/tmpzDhj9j''] {'quiet': False}
2020-10-05 15:33:39,931 - call returned (7, '')
2020-10-05 15:33:39,932 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:33:41 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:33:41,733 - call returned (255, '20/10/05 15:33:41 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:33:41,733 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp9bl4Fp 2>/tmp/tmp1ozRzZ''] {'quiet': False}
2020-10-05 15:33:41,795 - call returned (7, '')
2020-10-05 15:33:41,796 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:33:43 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:33:43,607 - call returned (255, '20/10/05 15:33:43 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:33:43,607 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:33:43,607 - Will retry 3 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:33:48,613 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpT4V98h 2>/tmp/tmpH7hScR''] {'quiet': False}
2020-10-05 15:33:48,673 - call returned (7, '')
2020-10-05 15:33:48,674 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:33:50 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:33:50,470 - call returned (255, '20/10/05 15:33:50 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:33:50,470 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp8jgNuM 2>/tmp/tmpKW9Pqx''] {'quiet': False}
2020-10-05 15:33:50,530 - call returned (7, '')
2020-10-05 15:33:50,530 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:33:52 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:33:52,329 - call returned (255, '20/10/05 15:33:52 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:33:52,329 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:33:52,329 - Will retry 2 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:33:57,335 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp2sC7N3 2>/tmp/tmplJYNfF''] {'quiet': False}
2020-10-05 15:33:57,394 - call returned (7, '')
2020-10-05 15:33:57,395 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:33:59 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:33:59,182 - call returned (255, '20/10/05 15:33:59 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:33:59,183 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpfl12tW 2>/tmp/tmpXyAXGZ''] {'quiet': False}
2020-10-05 15:33:59,247 - call returned (7, '')
2020-10-05 15:33:59,247 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:34:01 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:34:01,076 - call returned (255, '20/10/05 15:34:01 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:34:01,076 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:34:01,077 - Will retry 1 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:34:06,082 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp90J0QE 2>/tmp/tmpFGxjuV''] {'quiet': False}
2020-10-05 15:34:06,142 - call returned (7, '')
2020-10-05 15:34:06,142 - Getting jmx metrics from NN failed. URL: http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmp90J0QE 2>/tmp/tmpFGxjuV' returned 7. 

2020-10-05 15:34:06,143 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:34:07 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:34:07,946 - call returned (255, '20/10/05 15:34:07 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:34:07,947 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpz5GFSd 2>/tmp/tmpUzicMm''] {'quiet': False}
2020-10-05 15:34:08,007 - call returned (7, '')
2020-10-05 15:34:08,008 - Getting jmx metrics from NN failed. URL: http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmpz5GFSd 2>/tmp/tmpUzicMm' returned 7. 

2020-10-05 15:34:08,008 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:34:09 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:34:09,809 - call returned (255, '20/10/05 15:34:09 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:34:09,809 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:34:09,809 - No active NameNode was found after 5 retries. Will return current NameNode HA states
2020-10-05 15:34:09,809 - Will retry 94 time(s), caught exception: The NameNode nn1 is not listed as Active or Standby, waiting.... Sleeping for 5 sec(s)
2020-10-05 15:34:14,815 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpOOStV4 2>/tmp/tmpC2rhnS''] {'quiet': False}
2020-10-05 15:34:14,875 - call returned (7, '')
2020-10-05 15:34:14,875 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:34:16 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:34:16,670 - call returned (255, '20/10/05 15:34:16 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:34:16,670 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpIpp8Rl 2>/tmp/tmpxxcffl''] {'quiet': False}
2020-10-05 15:34:16,730 - call returned (7, '')
2020-10-05 15:34:16,731 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:34:18 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:34:18,538 - call returned (255, '20/10/05 15:34:18 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:34:18,538 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:34:18,538 - Will retry 4 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:34:23,544 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpyFXFE9 2>/tmp/tmpH8C3j6''] {'quiet': False}
2020-10-05 15:34:23,606 - call returned (7, '')
2020-10-05 15:34:23,606 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:34:25 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:34:25,462 - call returned (255, '20/10/05 15:34:25 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:34:25,462 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp56hPoz 2>/tmp/tmph0wT82''] {'quiet': False}
2020-10-05 15:34:25,523 - call returned (7, '')
2020-10-05 15:34:25,524 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:34:27 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:34:27,346 - call returned (255, '20/10/05 15:34:27 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:34:27,346 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:34:27,347 - Will retry 3 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:34:32,353 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpAUvBSg 2>/tmp/tmpawk7st''] {'quiet': False}
2020-10-05 15:34:32,412 - call returned (7, '')
2020-10-05 15:34:32,413 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:34:34 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:34:34,213 - call returned (255, '20/10/05 15:34:34 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:34:34,214 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpa80ZgK 2>/tmp/tmpCfE0Os''] {'quiet': False}
2020-10-05 15:34:34,274 - call returned (7, '')
2020-10-05 15:34:34,274 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:34:36 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:34:36,088 - call returned (255, '20/10/05 15:34:36 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:34:36,088 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:34:36,088 - Will retry 2 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:34:41,094 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmphrGbY5 2>/tmp/tmpy_CTY4''] {'quiet': False}
2020-10-05 15:34:41,159 - call returned (7, '')
2020-10-05 15:34:41,160 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:34:43 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:34:43,090 - call returned (255, '20/10/05 15:34:43 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:34:43,090 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpzWJ8rE 2>/tmp/tmpPCgHoH''] {'quiet': False}
2020-10-05 15:34:43,156 - call returned (7, '')
2020-10-05 15:34:43,157 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:34:44 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:34:44,991 - call returned (255, '20/10/05 15:34:44 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:34:44,991 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:34:44,991 - Will retry 1 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)
2020-10-05 15:34:49,994 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpkz_24b 2>/tmp/tmp6UZPeC''] {'quiet': False}
2020-10-05 15:34:50,055 - call returned (7, '')
2020-10-05 15:34:50,056 - Getting jmx metrics from NN failed. URL: http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmpkz_24b 2>/tmp/tmp6UZPeC' returned 7. 

2020-10-05 15:34:50,056 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:34:51 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:34:51,916 - call returned (255, '20/10/05 15:34:51 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:34:51,917 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpssn7Qd 2>/tmp/tmpbm1Fo7''] {'quiet': False}
2020-10-05 15:34:51,978 - call returned (7, '')
2020-10-05 15:34:51,979 - Getting jmx metrics from NN failed. URL: http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/jmx.py", line 38, in get_value_from_jmx
    _, data, _ = get_user_call_output(cmd, user=run_user, quiet=False)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output
    raise ExecutionFailed(err_msg, code, files_output[0], files_output[1])
ExecutionFailed: Execution of 'curl -s 'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem' 1>/tmp/tmpssn7Qd 2>/tmp/tmpbm1Fo7' returned 7. 

2020-10-05 15:34:51,979 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:34:53 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:34:53,847 - call returned (255, '20/10/05 15:34:53 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:34:53,847 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:34:53,848 - No active NameNode was found after 5 retries. Will return current NameNode HA states
2020-10-05 15:34:53,848 - Will retry 93 time(s), caught exception: The NameNode nn1 is not listed as Active or Standby, waiting.... Sleeping for 5 sec(s)
2020-10-05 15:34:58,854 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0328.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp7J99Tl 2>/tmp/tmpNXSIWs''] {'quiet': False}
2020-10-05 15:34:58,912 - call returned (7, '')
2020-10-05 15:34:58,913 - call['hdfs haadmin -ns TDHUAT -getServiceState nn1'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:35:00 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:35:00,715 - call returned (255, '20/10/05 15:35:00 INFO ipc.Client: Retrying connect to server: h8cc0328.stb.gov.sg/10.21.21.85:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0328.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:35:00,716 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://h8cc0329.stb.gov.sg:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpPTIbzY 2>/tmp/tmpp35ojo''] {'quiet': False}
2020-10-05 15:35:00,774 - call returned (7, '')
2020-10-05 15:35:00,775 - call['hdfs haadmin -ns TDHUAT -getServiceState nn2'] {'logoutput': True, 'user': 'hdfs'}
20/10/05 15:35:02 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)
Operation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
2020-10-05 15:35:02,572 - call returned (255, '20/10/05 15:35:02 INFO ipc.Client: Retrying connect to server: h8cc0329.stb.gov.sg/10.21.21.86:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)\nOperation failed: Call From H8CC0328.stb.gov.sg/10.21.21.85 to h8cc0329.stb.gov.sg:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused')
2020-10-05 15:35:02,572 - NameNode HA states: active_namenodes = [], standby_namenodes = [], unknown_namenodes = [(u'nn1', 'h8cc0328.stb.gov.sg:50070'), (u'nn2', 'h8cc0329.stb.gov.sg:50070')]
2020-10-05 15:35:02,572 - Will retry 4 time(s), caught exception: No active NameNode was found.. Sleeping for 5 sec(s)