CMS is down and cant able to get back that services

[#1]

Hello Team,

All of sudden service monitor was went down and when i checked the logs i can see connection refused error in logs.

I have tried to restart all the CMS services and this time all went down.

What i was did is i have deleted CMS and trying to re-add the CMS to cloudera manager,

This time i am not able to add the CMS and when i am trying to adding it active monitor and reports manager cant able to find databases and DB's in test connection stage.

Can some one please respond ASAP and it would be greatfull.

Thanks,

Vinod

[#2]

Hi Bgooley,

Thanks for response,

Find few more details below,

$ netstat -na | grep 7432
tcp 0 0 0.0.0.0:7432 0.0.0.0:* LISTEN
tcp 0 0 hostname:7432 hostname:51538 ESTABLISHED
tcp 0 0 hostname:7432 hostname:51786 ESTABLISHED
tcp 0 0 hostname:7432 hostname:51776 ESTABLISHED
tcp 0 0 hostname:51766 hostname:7432 ESTABLISHED
tcp 0 0 hostname:51542 hostname:7432 ESTABLISHED
tcp 0 0 hostname:7432 hostname:51778 ESTABLISHED
tcp 0 0 hostname:7432 hostname:51746 ESTABLISHED
tcp 0 0 hostname:7432 hostname:51758 ESTABLISHED
tcp 0 0 hostname:51776 hostname:7432 ESTABLISHED
tcp 0 0 hostname:7432 hostname:51754 ESTABLISHED
tcp 0 0 hostname:51536 hostname:7432 ESTABLISHED
tcp 0 0 hostname:7432 hostname:51788 ESTABLISHED
tcp 0 0 hostname:51772 hostname:7432 ESTABLISHED
tcp 0 0 hostname:51752 hostname:7432 ESTABLISHED
tcp 0 0 hostname:7432 hostname:51742 ESTABLISHED
tcp 0 0 hostname:51762 hostname:7432 ESTABLISHED
tcp 0 0 hostname:51756 hostname:7432 ESTABLISHED
tcp 0 0 hostname:7432 hostname:51790 ESTABLISHED
tcp 0 0 hostname:7432 hostname:34766 ESTABLISHED
tcp 0 0 hostname:34766 hostname:7432 ESTABLISHED
tcp 0 0 hostname:7432 hostname:51772 ESTABLISHED
tcp 0 0 hostname:51792 hostname:7432 ESTABLISHED
tcp 0 0 hostname:51750 hostname:7432 ESTABLISHED
tcp 0 0 hostname:7432 hostname:51539 ESTABLISHED
tcp 0 0 hostname:51784 hostname:7432 ESTABLISHED
tcp 0 0 hostname:7432 hostname:51782 ESTABLISHED
tcp 0 0 hostname:51774 hostname:7432 ESTABLISHED
tcp 0 0 hostname:7432 hostname:51760 ESTABLISHED
tcp 0 0 hostname:7432 hostname:51542 ESTABLISHED
tcp 0 0 hostname:51770 hostname:7432 ESTABLISHED
tcp 0 0 hostname:7432 hostname:51756 ESTABLISHED
tcp 0 0 hostname:7432 hostname:51764 ESTABLISHED
tcp 0 0 hostname:34768 hostname:7432 ESTABLISHED
tcp 0 0 hostname:51754 hostname:7432 ESTABLISHED
tcp 0 0 hostname:51768 hostname:7432 ESTABLISHED
tcp 0 0 hostname:7432 hostname:34764 ESTABLISHED
tcp 0 0 hostname:51544 hostname:7432 ESTABLISHED
tcp 0 0 hostname:7432 hostname:51752 ESTABLISHED
tcp 0 0 hostname:7432 hostname:51792 ESTABLISHED
tcp 0 0 hostname:51782 hostname:7432 ESTABLISHED
tcp 0 0 hostname:7432 hostname:34768 ESTABLISHED
tcp 0 0 hostname:51778 hostname:7432 ESTABLISHED
tcp 0 0 hostname:7432 hostname:51750 ESTABLISHED
tcp 0 0 hostname:51758 hostname:7432 ESTABLISHED
tcp 0 0 hostname:51538 hostname:7432 ESTABLISHED
tcp 0 0 hostname:7432 hostname:51770 ESTABLISHED
tcp 0 0 hostname:51744 hostname:7432 ESTABLISHED
tcp 0 0 hostname:51764 hostname:7432 ESTABLISHED
tcp 0 0 hostname:51790 hostname:7432 ESTABLISHED
tcp 0 0 hostname:51539 hostname:7432 ESTABLISHED
tcp 0 0 hostname:7432 hostname:51744 ESTABLISHED
tcp 0 0 hostname:34764 hostname:7432 ESTABLISHED
tcp 0 0 hostname:51760 hostname:7432 ESTABLISHED
tcp 0 0 hostname:7432 hostname:51766 ESTABLISHED
tcp 0 0 hostname:34770 hostname:7432 ESTABLISHED
tcp 0 0 hostname:51788 hostname:7432 ESTABLISHED
tcp 0 0 hostname:7432 hostname:51784 ESTABLISHED
tcp 0 0 hostname:7432 hostname:51536 ESTABLISHED
tcp 0 0 hostname:7432 hostname:51544 ESTABLISHED
tcp 0 0 hostname:7432 hostname:51762 ESTABLISHED
tcp 0 0 hostname:7432 hostname:34770 ESTABLISHED
tcp 0 0 hostname:51786 hostname:7432 ESTABLISHED
tcp 0 0 hostname:7432 hostname:51768 ESTABLISHED
tcp 0 0 hostname:51746 hostname:7432 ESTABLISHED
tcp 0 0 hostname:51742 hostname:7432 ESTABLISHED
tcp 0 0 hostname:7432 hostname:51774 ESTABLISHED
tcp 0 0 :::7432 :::* LISTEN
unix 2 [ ACC ] STREAM LISTENING 27666 /tmp/.s.PGSQL.7432
mcaf@tparhelmhp001:~:HADOOP PROD]$ netstat -na | grep 5432
tcp 0 0 127.0.0.1:5432 0.0.0.0:* LISTEN
tcp 0 0 hostname:54322 10.8.53.154:60020 TIME_WAIT
tcp 0 0 ::1:5432 :::* LISTEN
unix 2 [ ACC ] STREAM LISTENING 27649 /tmp/.s.PGSQL.5432

As you said if i am trying to add CMS to cloudera automatically i can see Activity and Reports manager in list, how can i remove them in a list ?

And i can see below error's in stderr.log for acivity monitor,

log4j:ERROR Could not find value for key log4j.appender.A
log4j:ERROR Could not instantiate appender named "A".

And i can see below error's in stderr.log for reports manager,

log4j:ERROR Could not find value for key log4j.appender.A
log4j:ERROR Could not instantiate appender named "A".

Please revert me back.

Best Regards,

Vinod

[#3]

@kvinod 

For sure you have a double entry in your db.mgmt.properties  one simple way is to split the files as I have done below  [------]   

Create a back of the current file cp /etc/cloudera-scm-server/db.mgmt.properties  /etc/cloudera-scm-server/db.mgmt.properties.ORIG

Then override the db.mgmt.properties with first the  ORANGE if nothing works then the BLUE

$ sudo vi /etc/cloudera-scm-server/db.mgmt.properties
# The source of truth for these settings
# is the Cloudera Manager databases and
# changes made here will not be reflected
# there automatically.
#
com.cloudera.cmf.ACTIVITYMONITOR.db.type=postgresql
com.cloudera.cmf.ACTIVITYMONITOR.db.host=hostname:7432
com.cloudera.cmf.ACTIVITYMONITOR.db.name=amon
com.cloudera.cmf.ACTIVITYMONITOR.db.user=amon
com.cloudera.cmf.ACTIVITYMONITOR.db.password=4WB4R5yxnp
com.cloudera.cmf.REPORTSMANAGER.db.type=postgresql
com.cloudera.cmf.REPORTSMANAGER.db.host=hostname:7432
com.cloudera.cmf.REPORTSMANAGER.db.name=rman
com.cloudera.cmf.REPORTSMANAGER.db.user=rman
com.cloudera.cmf.REPORTSMANAGER.db.password=WceGeruLNG
com.cloudera.cmf.NAVIGATOR.db.type=postgresql
com.cloudera.cmf.NAVIGATOR.db.host=hostname:7432
com.cloudera.cmf.NAVIGATOR.db.name=nav
com.cloudera.cmf.NAVIGATOR.db.user=nav
com.cloudera.cmf.NAVIGATOR.db.password=D2tjw5xjoE
com.cloudera.cmf.NAVIGATORMETASERVER.db.type=postgresql
com.cloudera.cmf.NAVIGATORMETASERVER.db.host=hostname:7432
com.cloudera.cmf.NAVIGATORMETASERVER.db.name=navms
com.cloudera.cmf.NAVIGATORMETASERVER.db.user=navms
com.cloudera.cmf.NAVIGATORMETASERVER.db.password=elJRINTAth

[--------------------------separator-----------------------------]

# The source of truth for these settings
# is the Cloudera Manager databases and
# changes made here will not be reflected
# there automatically.
#
com.cloudera.cmf.ACTIVITYMONITOR.db.type=postgresql
com.cloudera.cmf.ACTIVITYMONITOR.db.host=hostname:7432
com.cloudera.cmf.ACTIVITYMONITOR.db.name=amon
com.cloudera.cmf.ACTIVITYMONITOR.db.user=amon
com.cloudera.cmf.ACTIVITYMONITOR.db.password=O31A60K5SN
com.cloudera.cmf.REPORTSMANAGER.db.type=postgresql
com.cloudera.cmf.REPORTSMANAGER.db.host=hostname:7432
com.cloudera.cmf.REPORTSMANAGER.db.name=rman
com.cloudera.cmf.REPORTSMANAGER.db.user=rman
com.cloudera.cmf.REPORTSMANAGER.db.password=BPPShP0O9k
com.cloudera.cmf.NAVIGATOR.db.type=postgresql
com.cloudera.cmf.NAVIGATOR.db.host=hostname:7432
com.cloudera.cmf.NAVIGATOR.db.name=nav
com.cloudera.cmf.NAVIGATOR.db.user=nav
com.cloudera.cmf.NAVIGATOR.db.password=QHYL7zUSQe
com.cloudera.cmf.NAVIGATORMETASERVER.db.type=postgresql
com.cloudera.cmf.NAVIGATORMETASERVER.db.host=hostname:7432
com.cloudera.cmf.NAVIGATORMETASERVER.db.name=navms
com.cloudera.cmf.NAVIGATORMETASERVER.db.user=navms
com.cloudera.cmf.NAVIGATORMETASERVER.db.password=elJRINTAth

After testing the 2 options depending on success or failure then we could get another option. Make sure you have also followed these steps link. 

https://www.cloudera.com/documentation/enterprise/5-14-x/topics/cm_ig_extrnl_pstgrs.html#cmig_topic_...

Please revert 

[#4]

Hello Shelton,

Thanks for your reply...!

I have tried as you suggested but the thing is am seeing the same issue not even its related to password mismatch.

If any password related mismatch may be we can do something else to fix it.

Still facing same issue.

Any suggestions are appreciated.

Thanks,

Vinod

[#5]

@kvinod 

Is it a production cluster or not? If its the latter is there a reason for not using some other database like Mysql/MariaDB/Oracle?

In my implementations I tend to favour these 3 hence I have more hands-on in your case you could try out MariaDB/Mysql as they are open source 

[#6]

@kvinod 

You can use MySQL for the component please follow 

https://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_mysql.html

[#7 · Accepted solution]

Hello Team,

Sorry for late response, I have created external metastore using MySql and make them up. Now my cluster is looking good.

Thanks Team for your efforts .

Best Regards,

Vinod

[#8]

@kvinod 

Good then close the thread. It's always a good idea to have an external database to host not only the CM/Ambari database but also Oozie, hive, ranger etc in a multitenant architecture and maybe backup the database host periodically to avoid having your backend databases all over the cluster.

20 cents advice !