Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Service Monitor and Host Monitor DNS name corrupted?

Highlighted

Service Monitor and Host Monitor DNS name corrupted?

New Contributor

Hi, I have been logged out of my CM for about a week, upon logging back in I found my Host and Service monitor asking for a stale config restart, and the diff showed that the first letter of the host name for the service monitor had been removed!

 

I'm in AWS, and for example the diff showed that the "i" in my internal AWS host name was removed. I tried to not accept the config change, but I think it's already made because now I'm getting these log entries (see below) note the host name, with a missing leading "i".

 

All of the other services and the node itself are addressed (internally in the VPC) as "ip-172-30-23-73.us-east-2.compute.internal" and are functioning normally with that DNS name. Can anyone describe how to find this config and fix the DNS name by re-adding the leading "i" OR remove the Service/Host monitor roles and re-add them? Thanks.

 

====

 

Failed to publish event: SimpleEvent{attributes={STACKTRACE=[java.net.UnknownHostException: p-172-30-23-73.us-east-2.compute.internal at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:178) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at sun.net.NetworkClient.doConnect(NetworkClient.java:175) at sun.net.www.http.HttpClient.openServer(HttpClient.java:432) at sun.net.www.http.HttpClient.openServer(HttpClient.java:527) at sun.net.www.http.HttpClient.(HttpClient.java:211) at sun.net.www.http.HttpClient.New(HttpClient.java:308) at sun.net.www.http.HttpClient.New(HttpClient.java:326) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:996) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:932) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:850) at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1091) at com.cloudera.cmf.BasicScmProxy.authenticate(BasicScmProxy.java:276) at com.cloudera.cmf.BasicScmProxy.fetch(BasicScmProxy.java:596) at com.cloudera.cmf.BasicScmProxy.getFragmentAndHash(BasicScmProxy.java:686) at com.cloudera.cmf.DescriptorAndFragments.newDescriptorAndFragments(DescriptorAndFragments.java:64) at com.cloudera.cmon.firehose.Main.main(Main.java:392) ], EXCEPTION_TYPES=[java.net.UnknownHostException], ROLE=[mgmt-SERVICEMONITOR-63312b00756c55af1182743c58b027d4], SEVERITY=[IMPORTANT], SERVICE=[mgmt], HOST_IDS=[f0b0fe3a-22f9-431c-829e-52f56b8695b4], LOG_LEVEL=[WARN], ROLE_TYPE=[SERVICEMONITOR], CATEGORY=[LOG_MESSAGE], SERVICE_TYPE=[MGMT], HOSTS=[ip-172-31-21-191.us-east-2.compute.internal], EVENTCODE=[EV_LOG_EVENT]}, content=Exception while getting fetch configDefaults hash: none, timestamp=1544805519191}

2 REPLIES 2

Re: Service Monitor and Host Monitor DNS name corrupted?

Expert Contributor

Hi,

 

This is quite unusual, the configuration normally doesn't change like this. Can you please login to Cloudera Manager then goto the follow location and check to see if it's been set to something unexepcted.

 

CM -> Administration -> Settings -> Cloudera Manager Hostname Override

 

If the value is blank, which is the default, it may indicate that the result of InetAddress.getLocalhost() is incorrect which can be caused by a number of things including entries in /etc/hosts. If you are certain that DNS works properly and that there are no erronous entries in /etc/hosts you can try setting the HostName Override. Then restart both Cloudera Manager and the Management services.

---
Customer Operations Engineer | Security SME | Cloudera, Inc.
Highlighted

Re: Service Monitor and Host Monitor DNS name corrupted?

New Contributor

Hello 'lhebert' - 

 

Thanks for your quick reply - I got into the remote server who was showing the problematic conneciton shown in the log and cracked open the ...../cmon.log specified in the log trail and found this:

 

(I'm mangling the host name for security reasons but you'll see the problem)

 

<property>

    <name>scm.server.url</name>

    <value>http://p-190-13-27-23.us-east-2.compute.internal:7180</value>

  </property>

 

...and this...

 

eventcatcher.scm.server.url=http://p-190-13-27-23.us-east-2.compute.internal:7180

 

I'm going to attempt a manual repair... will report back.

Don't have an account?
Coming from Hortonworks? Activate your account here