Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Failed to run DnsTest

Highlighted

Failed to run DnsTest

New Contributor

Hi all,

 

I am getting below error in the logs. Could you please help?

 

[08/Mar/2017 16:31:20 +0000] 8842 DnsResolutionMonitor throttling_logger ERROR Failed to run DnsTest.
Traceback (most recent call last):
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.7.1-py2.6.egg/cmf/monitor/host/dns_names.py", line 83, in collect_dns_metrics
self._subprocess_with_timeout(args, self._poll_timeout)
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.7.1-py2.6.egg/cmf/monitor/host/dns_names.py", line 55, in _subprocess_with_timeout
return subprocess_with_timeout(args, timeout)
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.7.1-py2.6.egg/cmf/subprocess_timeout.py", line 94, in subprocess_with_timeout
raise Exception("timeout with args %s" % args)
Exception: timeout with args ['/usr/java/jdk1.7.0_67-cloudera/bin/java', '-classpath', '/usr/share/cmf/lib/agent-5.7.1.jar', 'com.cloudera.cmon.agent.DnsTest']
[08/Mar/2017 16:31:21 +0000] 8842 Monitor-GenericMonitor throttling_logger WARNING (11 skipped) Error collecting metrics from sample file. Result: 4
[08/Mar/2017 16:31:21 +0000] 8842 Monitor-HostMonitor filesystem_map WARNING Failed to join worker process collecting filesystem usage. All nodev filesystems will have unknown usage until the worker process is no longer active. Current nodev filesystems: /dev/shm,/var/run/cloudera-scm-agent/process
[08/Mar/2017 16:35:35 +0000] 8842 MainThread heartbeat_tracker INFO HB stats (seconds): num:25 LIFE_MIN:0.02 min:0.06 mean:0.07 max:0.09 LIFE_MAX:1.56
[08/Mar/2017 16:35:35 +0000] 8842 MainThread throttling_logger INFO (12 skipped) Identified java component java7 with full version JAVA_HOME=/usr/java/default java version "1.7.0_71" Java(TM) SE Runtime Environment (build 1.7.0_71-b14) Java HotSpot(TM) 64-Bit Server VM (build 24.71-b01, mixed mode) for requested version misc.
[08/Mar/2017 16:35:35 +0000] 8842 MainThread throttling_logger INFO (12 skipped) Identified java component java7 with full version JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera java version "1.7.0_67" Java(TM) SE Runtime Environment (build 1.7.0_67-b01) Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode) for requested version 7.
[08/Mar/2017 16:45:36 +0000] 8842 MainThread heartbeat_tracker INFO HB stats (seconds): num:41 LIFE_MIN:0.02 min:0.06 mean:0.07 max:0.09 LIFE_MAX:1.56
[08/Mar/2017 16:55:37 +0000] 8842 MainThread heartbeat_tracker INFO HB stats (seconds): num:40 LIFE_MIN:0.02 min:0.06 mean:0.07 max:0.08 LIFE_MAX:1.56
[08/Mar/2017 17:05:37 +0000] 8842 MainThread heartbeat_tracker INFO HB stats (seconds): num:40 LIFE_MIN:0.02 min:0.06 mean:0.07 max:0.08 LIFE_MAX:1.56
[08/Mar/2017 17:05:37 +0000] 8842 MainThread throttling_logger INFO (14 skipped) Identified java component java7 with full version JAVA_HOME=/usr/java/default java version "1.7.0_71" Java(TM) SE Runtime Environment (build 1.7.0_71-b14) Java HotSpot(TM) 64-Bit Server VM (build 24.71-b01, mixed mode) for requested version misc.
[08/Mar/2017 17:05:37 +0000] 8842 MainThread throttling_logger INFO (14 skipped) Identified java component java7 with full version JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera java version "1.7.0_67" Java(TM) SE Runtime Environment (build 1.7.0_67-b01) Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode) for requested version 7.
[08/Mar/2017 17:15:38 +0000] 8842 MainThread heartbeat_tracker INFO HB stats (seconds): num:40 LIFE_MIN:0.02 min:0.06 mean:0.07 max:0.08 LIFE_MAX:1.56
[08/Mar/2017 17:25:38 +0000] 8842 MainThread heartbeat_tracker INFO HB stats (seconds): num:40 LIFE_MIN:0.02 min:0.06 mean:0.07 max:0.08 LIFE_MAX:1.56
[08/Mar/2017 17:35:39 +0000] 8842 MainThread heartbeat_tracker INFO HB stats (seconds): num:40 LIFE_MIN:0.02 min:0.06 mean:0.07 max:0.08 LIFE_MAX:1.56
[08/Mar/2017 17:35:39 +0000] 8842 MainThread throttling_logger INFO (14 skipped) Identified java component java7 with full version JAVA_HOME=/usr/java/default java version "1.7.0_71" Java(TM) SE Runtime Environment (build 1.7.0_71-b14) Java HotSpot(TM) 64-Bit Server VM (build 24.71-b01, mixed mode) for requested version misc.
[08/Mar/2017 17:35:39 +0000] 8842 MainThread throttling_logger INFO (14 skipped) Identified java component java7 with full version JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera java version "1.7.0_67" Java(TM) SE Runtime Environment (build 1.7.0_67-b01) Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode) for requested version 7.

4 REPLIES 4

Re: Failed to run DnsTest

New Contributor

I already checked /etc/hosts on all the 4 nodes. It has hostname set with FQDN.

Re: Failed to run DnsTest

Super Guru

@PranavAWagde,

 

The stack shows the command that the agent used to try to run the DNS health check:

 

Exception: timeout with args ['/usr/java/jdk1.7.0_67-cloudera/bin/java', '-classpath', '/usr/share/cmf/lib/agent-5.7.1.jar', 'com.cloudera.cmon.agent.DnsTest']

 

The command did not complete within 2 seconds.  In Cloudera Manager 5.10, we wait 30 seconds before timing out.

 

You can observe what happens by running the following on your agent host where the issue is occurring:

/usr/java/jdk1.7.0_67-cloudera/bin/java -classpath /usr/share/cmf/lib/agent-5.7.1.jar com.cloudera.cmon.agent.DnsTest

 

That might clarify how long the command takes to return.  It can fluctuate sometimes.

 

- Ben

 

Re: Failed to run DnsTest

New Contributor

Hi Ben,

 

Thanks for the reply.

 

I have 4 node cluster and below is the output of the java program(changed the hostname, to avoid exposing customer name) 

 

Each program executed in less than 2 seconds.

 

-linx-011:~# /usr/java/jdk1.7.0_67-cloudera/bin/java -classpath /usr/share/cmf/lib/agent-5.7.1.jar com.cloudera.cmon.agent.DnsTest
{"status": "0", "ip": "10.197.45.14", "hostname": "linx-011.example.com", "canonicalname": "linx-011.example.com", "localhostDuration": "5", "canonicalnameDuration": "0" }

-linx-012:~# /usr/java/jdk1.7.0_67-cloudera/bin/java -classpath /usr/share/cmf/lib/agent-5.7.1.jar com.cloudera.cmon.agent.DnsTest
{"status": "0", "ip": "10.197.45.15", "hostname":"linx-012.example.com", "canonicalname": "linx-012.example.com","localhostDuration": "5", "canonicalnameDuration": "0" }

-linx-013:~# /usr/java/jdk1.7.0_67-cloudera/bin/java -classpath /usr/share/cmf/lib/agent-5.7.1.jar com.cloudera.cmon.agent.DnsTest
{"status": "0", "ip": "10.197.45.16", "hostname":"linx-013.example.com", "canonicalname": "linx-013.example.com", "localhostDuration": "5", "canonicalnameDuration": "1" }

-linx-014:~# /usr/java/jdk1.7.0_67-cloudera/bin/java -classpath /usr/share/cmf/lib/agent-5.7.1.jar com.cloudera.cmon.agent.DnsTest
{"status": "0", "ip": "10.197.45.17", "hostname":"linx-014.example.com", "canonicalname": "linx-014.example.com", "localhostDuration": "6", "canonicalnameDuration": "0" }

Re: Failed to run DnsTest

New Contributor

Forgot to add, we have cloudera version as 5.7.1

Don't have an account?
Coming from Hortonworks? Activate your account here