Reply
New Contributor
Posts: 5
Registered: ‎04-07-2017

Failed to run DnsTest

Hi all,

 

I am getting below error in the logs. Could you please help?

 

[08/Mar/2017 16:31:20 +0000] 8842 DnsResolutionMonitor throttling_logger ERROR Failed to run DnsTest.
Traceback (most recent call last):
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.7.1-py2.6.egg/cmf/monitor/host/dns_names.py", line 83, in collect_dns_metrics
self._subprocess_with_timeout(args, self._poll_timeout)
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.7.1-py2.6.egg/cmf/monitor/host/dns_names.py", line 55, in _subprocess_with_timeout
return subprocess_with_timeout(args, timeout)
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.7.1-py2.6.egg/cmf/subprocess_timeout.py", line 94, in subprocess_with_timeout
raise Exception("timeout with args %s" % args)
Exception: timeout with args ['/usr/java/jdk1.7.0_67-cloudera/bin/java', '-classpath', '/usr/share/cmf/lib/agent-5.7.1.jar', 'com.cloudera.cmon.agent.DnsTest']
[08/Mar/2017 16:31:21 +0000] 8842 Monitor-GenericMonitor throttling_logger WARNING (11 skipped) Error collecting metrics from sample file. Result: 4
[08/Mar/2017 16:31:21 +0000] 8842 Monitor-HostMonitor filesystem_map WARNING Failed to join worker process collecting filesystem usage. All nodev filesystems will have unknown usage until the worker process is no longer active. Current nodev filesystems: /dev/shm,/var/run/cloudera-scm-agent/process
[08/Mar/2017 16:35:35 +0000] 8842 MainThread heartbeat_tracker INFO HB stats (seconds): num:25 LIFE_MIN:0.02 min:0.06 mean:0.07 max:0.09 LIFE_MAX:1.56
[08/Mar/2017 16:35:35 +0000] 8842 MainThread throttling_logger INFO (12 skipped) Identified java component java7 with full version JAVA_HOME=/usr/java/default java version "1.7.0_71" Java(TM) SE Runtime Environment (build 1.7.0_71-b14) Java HotSpot(TM) 64-Bit Server VM (build 24.71-b01, mixed mode) for requested version misc.
[08/Mar/2017 16:35:35 +0000] 8842 MainThread throttling_logger INFO (12 skipped) Identified java component java7 with full version JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera java version "1.7.0_67" Java(TM) SE Runtime Environment (build 1.7.0_67-b01) Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode) for requested version 7.
[08/Mar/2017 16:45:36 +0000] 8842 MainThread heartbeat_tracker INFO HB stats (seconds): num:41 LIFE_MIN:0.02 min:0.06 mean:0.07 max:0.09 LIFE_MAX:1.56
[08/Mar/2017 16:55:37 +0000] 8842 MainThread heartbeat_tracker INFO HB stats (seconds): num:40 LIFE_MIN:0.02 min:0.06 mean:0.07 max:0.08 LIFE_MAX:1.56
[08/Mar/2017 17:05:37 +0000] 8842 MainThread heartbeat_tracker INFO HB stats (seconds): num:40 LIFE_MIN:0.02 min:0.06 mean:0.07 max:0.08 LIFE_MAX:1.56
[08/Mar/2017 17:05:37 +0000] 8842 MainThread throttling_logger INFO (14 skipped) Identified java component java7 with full version JAVA_HOME=/usr/java/default java version "1.7.0_71" Java(TM) SE Runtime Environment (build 1.7.0_71-b14) Java HotSpot(TM) 64-Bit Server VM (build 24.71-b01, mixed mode) for requested version misc.
[08/Mar/2017 17:05:37 +0000] 8842 MainThread throttling_logger INFO (14 skipped) Identified java component java7 with full version JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera java version "1.7.0_67" Java(TM) SE Runtime Environment (build 1.7.0_67-b01) Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode) for requested version 7.
[08/Mar/2017 17:15:38 +0000] 8842 MainThread heartbeat_tracker INFO HB stats (seconds): num:40 LIFE_MIN:0.02 min:0.06 mean:0.07 max:0.08 LIFE_MAX:1.56
[08/Mar/2017 17:25:38 +0000] 8842 MainThread heartbeat_tracker INFO HB stats (seconds): num:40 LIFE_MIN:0.02 min:0.06 mean:0.07 max:0.08 LIFE_MAX:1.56
[08/Mar/2017 17:35:39 +0000] 8842 MainThread heartbeat_tracker INFO HB stats (seconds): num:40 LIFE_MIN:0.02 min:0.06 mean:0.07 max:0.08 LIFE_MAX:1.56
[08/Mar/2017 17:35:39 +0000] 8842 MainThread throttling_logger INFO (14 skipped) Identified java component java7 with full version JAVA_HOME=/usr/java/default java version "1.7.0_71" Java(TM) SE Runtime Environment (build 1.7.0_71-b14) Java HotSpot(TM) 64-Bit Server VM (build 24.71-b01, mixed mode) for requested version misc.
[08/Mar/2017 17:35:39 +0000] 8842 MainThread throttling_logger INFO (14 skipped) Identified java component java7 with full version JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera java version "1.7.0_67" Java(TM) SE Runtime Environment (build 1.7.0_67-b01) Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode) for requested version 7.

New Contributor
Posts: 5
Registered: ‎04-07-2017

Re: Failed to run DnsTest

I already checked /etc/hosts on all the 4 nodes. It has hostname set with FQDN.

Posts: 1,042
Topics: 1
Kudos: 262
Solutions: 130
Registered: ‎04-22-2014

Re: Failed to run DnsTest

@PranavAWagde,

 

The stack shows the command that the agent used to try to run the DNS health check:

 

Exception: timeout with args ['/usr/java/jdk1.7.0_67-cloudera/bin/java', '-classpath', '/usr/share/cmf/lib/agent-5.7.1.jar', 'com.cloudera.cmon.agent.DnsTest']

 

The command did not complete within 2 seconds.  In Cloudera Manager 5.10, we wait 30 seconds before timing out.

 

You can observe what happens by running the following on your agent host where the issue is occurring:

/usr/java/jdk1.7.0_67-cloudera/bin/java -classpath /usr/share/cmf/lib/agent-5.7.1.jar com.cloudera.cmon.agent.DnsTest

 

That might clarify how long the command takes to return.  It can fluctuate sometimes.

 

- Ben

 

New Contributor
Posts: 5
Registered: ‎04-07-2017

Re: Failed to run DnsTest

Hi Ben,

 

Thanks for the reply.

 

I have 4 node cluster and below is the output of the java program(changed the hostname, to avoid exposing customer name) 

 

Each program executed in less than 2 seconds.

 

-linx-011:~# /usr/java/jdk1.7.0_67-cloudera/bin/java -classpath /usr/share/cmf/lib/agent-5.7.1.jar com.cloudera.cmon.agent.DnsTest
{"status": "0", "ip": "10.197.45.14", "hostname": "linx-011.example.com", "canonicalname": "linx-011.example.com", "localhostDuration": "5", "canonicalnameDuration": "0" }

-linx-012:~# /usr/java/jdk1.7.0_67-cloudera/bin/java -classpath /usr/share/cmf/lib/agent-5.7.1.jar com.cloudera.cmon.agent.DnsTest
{"status": "0", "ip": "10.197.45.15", "hostname":"linx-012.example.com", "canonicalname": "linx-012.example.com","localhostDuration": "5", "canonicalnameDuration": "0" }

-linx-013:~# /usr/java/jdk1.7.0_67-cloudera/bin/java -classpath /usr/share/cmf/lib/agent-5.7.1.jar com.cloudera.cmon.agent.DnsTest
{"status": "0", "ip": "10.197.45.16", "hostname":"linx-013.example.com", "canonicalname": "linx-013.example.com", "localhostDuration": "5", "canonicalnameDuration": "1" }

-linx-014:~# /usr/java/jdk1.7.0_67-cloudera/bin/java -classpath /usr/share/cmf/lib/agent-5.7.1.jar com.cloudera.cmon.agent.DnsTest
{"status": "0", "ip": "10.197.45.17", "hostname":"linx-014.example.com", "canonicalname": "linx-014.example.com", "localhostDuration": "6", "canonicalnameDuration": "0" }

Highlighted
New Contributor
Posts: 5
Registered: ‎04-07-2017

Re: Failed to run DnsTest

Forgot to add, we have cloudera version as 5.7.1

Announcements