Reply
New Contributor
Posts: 3
Registered: ‎04-11-2019

Some of the services in cloud era manager not listening on their respective port

Hi Guys,

 

My cloud era runs on RHEL 7 box with CDH 6. I might have killed few processes by mistake while trying to do something. When I tried checking netstat of all ports related to cloudera Manager and Cloudera Navigator it looks llike below:

netstat -tulpna | grep LISTEN | grep 7180
tcp 0 0 0.0.0.0:7180 0.0.0.0:* LISTEN 11521/java

netstat -tulpna | grep LISTEN | grep 7183

netstat -tulpna | grep LISTEN | grep 7182
tcp 0 0 0.0.0.0:7182 0.0.0.0:* LISTEN 11521/java

netstat -tulpna | grep LISTEN | grep 7432

netstat -tulpna | grep LISTEN | grep 9000
tcp 0 0 10.11.65.99:9000 0.0.0.0:* LISTEN 14313/python2

netstat -tulpna | grep LISTEN | grep 19001
tcp 0 0 127.0.0.1:19001 0.0.0.0:* LISTEN 14124/python2

netstat -tulpna | grep LISTEN | grep 7184
tcp 0 0 0.0.0.0:7184 0.0.0.0:* LISTEN 43921/java

netstat -tulpna | grep LISTEN | grep 7185
tcp 0 0 0.0.0.0:7185 0.0.0.0:* LISTEN 43921/java

netstat -tulpna | grep LISTEN | grep 8084
tcp 0 0 0.0.0.0:8084 0.0.0.0:* LISTEN 43921/java

netstat -tulpna | grep LISTEN | grep 10101
tcp 0 0 0.0.0.0:10101 0.0.0.0:* LISTEN 44129/java

netstat -tulpna | grep LISTEN | grep 8086
tcp 0 0 10.11.65.99:8086 0.0.0.0:* LISTEN 43922/java

netstat -tulpna | grep LISTEN | grep 9997
tcp 0 0 10.11.65.99:9997 0.0.0.0:* LISTEN 43922/java

netstat -tulpna | grep LISTEN | grep 9996
tcp 0 0 0.0.0.0:9996 0.0.0.0:* LISTEN 43922/java

netstat -tulpna | grep LISTEN | grep 8087
tcp 0 0 10.11.65.99:8087 0.0.0.0:* LISTEN 43923/java

netstat -tulpna | grep LISTEN | grep 9999
tcp 0 0 10.11.65.99:9999 0.0.0.0:* LISTEN 43923/java

netstat -tulpna | grep LISTEN | grep 9998
tcp 0 0 0.0.0.0:9998 0.0.0.0:* LISTEN 43923/java

netstat -tulpna | grep LISTEN | grep 8091
tcp 0 0 10.11.65.99:8091 0.0.0.0:* LISTEN 43920/java

netstat -tulpna | grep LISTEN | grep 9995
tcp 0 0 10.11.65.99:9995 0.0.0.0:* LISTEN 43920/java

netstat -tulpna | grep LISTEN | grep 9994
tcp 0 0 0.0.0.0:9994 0.0.0.0:* LISTEN 43920/java

netstat -tulpna | grep LISTEN | grep 5678

netstat -tulpna | grep LISTEN | grep 8083

netstat -tulpna | grep LISTEN | grep 7186

netstat -tulpna | grep LISTEN | grep 7187

netstat -tulpna | grep LISTEN | grep 4867

netstat -tulpna | grep LISTEN | grep 7180
tcp 0 0 0.0.0.0:7180 0.0.0.0:* LISTEN 11521/java

netstat -tulpna | grep LISTEN | grep 8020
tcp 0 0 10.11.65.99:8020 0.0.0.0:* LISTEN 25858/java

netstat -tulpna | grep LISTEN | grep 50010

 

As you can see some ports are not listening. What would you suggest me to do? 

Thank you

 

Posts: 994
Kudos: 31
Solutions: 20
Registered: ‎05-27-2014

Re: Some of the services in cloud era manager not listening on their respective port

Hi @Navyaa ,

 

You can refer to this documentation link to refer to the ports used by CM and Navigator:

https://www.cloudera.com/documentation/enterprise/6/6.2/topics/cm_ig_ports_cm.html#cm_cn_ports

 

So from the output you sent, it looks like below services are down:

- Reports Manager

- Navigator Metadata Server

- Navigator Audit Server

 

From Cloudera Manager UI, do you find above three services reporting issues? Have you tried to do a stop and then start of the services? If you have not tried yet, I would recommend you to do that from Cloudera Manager UI.

 

Thanks,

Li

Li Wang, Technical Resolution Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

Terms of Service

Community Guidelines

How to use the forum

New Contributor
Posts: 3
Registered: ‎04-11-2019

Re: Some of the services in cloud era manager not listening on their respective port

[ Edited ]

Hi @lwang ,

 

Thank you for the reply. I did refer to the ports from the link you shared before.

 

Bad health reported for Activity Monitor Health, Service Monitor Health, Host Monitor Health, Event Server Health, Alert Publisher Health. I have stopped and started complete cluster, restarted cluster, stopped and started Cloud manager service, restarted Cloud manager service, stopped and started each service seperately. But nothing has really helped. After each service is started it again says : 

1. HBase - Bad : This role's process exited. This role is supposed to be started.

2. HBase - Bad : Master summary: entrada2.iedr.ie (Availability: Unknown, Health: Bad). This health test is bad because the Service Monitor did not find an active Master.

3. Hue server - The health of this role's host is bad. The following health tests are bad: clock offset. The following health tests are concerning: swapping, agent parcel directory.

4. Hue , Oozie, Impala, Key-Value Store Indexer,  Solr, Zoo Keeper - Healthy Hue Server: 0. Concerning Hue Server: 0. Total Hue Server: 1. Percent healthy: 0.00%. Percent healthy or concerning: 0.00%. Critical threshold: 51.00%.

5. Spark - Bad : This role's process exited. This role is supposed to be started.

6. YARN - Bad : ResourceManager summary: entrada2.iedr.ie (Availability: Active, Health: Bad). This health test reflects the health of the active ResourceManager.

 

On Hue editor when running any query it says - Invalid method name: 'ExecuteStatement' (code THRIFTAPPLICATION): None

 

And also Clock offset is throwing error: The host's NTP service could not be located or did not respond to a request for the clock offset. But I am using chronyc instead of NTP.

 

 

Thank you.

 

Cloudera Employee
Posts: 229
Registered: ‎01-15-2015

Re: Some of the services in cloud era manager not listening on their respective port

If you switched from using ntp to using chrony service then please restart the CM agent on all cluster nodes. This should help with the clock offset related issues.

 

For roles with health alert This role's process exited please review the stdout, stderr and role logs to find out details why the startup failed, those are found in /var/run/cloudera-scm-agent/process directory, lookup e.g. the recent directories for Spark and review the stdout/stderr logs for error messages.

Highlighted
New Contributor
Posts: 3
Registered: ‎04-11-2019

Re: Some of the services in cloud era manager not listening on their respective port

Hi @gzigldrum ,

 

Thank you for the reply. 

 

I did have a look at stderr.log to see errors such as  - java.io.IOException: No space left on device.

Will address those errors.

 

Can you please let me know where do I have to look for switching from using ntp to using chrony service? 

 

Thanks a Mil.