Reply
Explorer
Posts: 13
Registered: ‎10-28-2013

Get Cloudera Manager back to good health

Hello Everybody,

After installing Cloudera Manager successfully on a two node cluster I encountered a period of inactivity whilst I replaced a failed host node.

However now my Cloudera Manager's health has gone from good to bad. Oozie1, Hue1, and Hive1 show bad health and all the others show unknown health.

The configuration issues for my Cluster all contain the following message: No host heartbeat; CDH versions cannot be verified.

I have attached the Configuration Issues file below.


When I try the hbase or hdfs Web UI, etc I get the error message: Firefox can't establish a connection to the server at masternode:60010 or masternode:50070.

Also no Charts are working (but this has been since installation).

I am due to add two new hosts soon.

How do get Cloudera Manager back to good health ?

 

Thanks,

Francesco

 

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

 

Charts

    * Internal error while querying the Host Monitor
    * Unable to issue query: request to the Host Monitor timed out


Bad Health
oozie1 , hive1 , hue1

Configuration issues

hbase1: Configuration Issues

    * regionserver (masternode): No host heartbeat; CDH versions cannot be verified.
    * master (masternode): No host heartbeat; CDH versions cannot be verified.

hdfs1: Configuration Issues

    * hdfs1: Service hdfs1 has 1 DataNode. Cloudera suggests at least 3 DataNodes for HDFS.
    * hdfs1: Java Heap Size of Namenode in Bytes
      Java Heap Size of Namenode in Bytes is recommended to be at least 1GB for every million HDFS blocks. Suggested minimum value: 1073741824
    * secondarynamenode (masternode): No host heartbeat; CDH versions cannot be verified.
    * namenode (masternode): No host heartbeat; CDH versions cannot be verified.
    * balancer (masternode): No host heartbeat; CDH versions cannot be verified.
    * datanode (masternode): No host heartbeat; CDH versions cannot be verified.

hive1: Configuration Issues

    * gateway (masternode): No host heartbeat; CDH versions cannot be verified.
    * hivemetastore (masternode): No host heartbeat; CDH versions cannot be verified.

hue1: Configuration Issues

    * beeswax_server (masternode): No host heartbeat; CDH versions cannot be verified.
    * hue_server (masternode): No host heartbeat; CDH versions cannot be verified.

mapreduce1: Configuration Issues

    * mapreduce1: I/O Sort Memory Buffer (MiB)
      I/O Sort Memory Buffer (MiB) is recommended to be between 25% to 70% of the value of MapReduce Child Java Maximum Heap Size. Suggested range: [16‥41]
    * tasktracker (masternode): No host heartbeat; CDH versions cannot be verified.
    * jobtracker (masternode): No host heartbeat; CDH versions cannot be verified.

oozie1: Configuration Issues

    * oozie_server (masternode): No host heartbeat; CDH versions cannot be verified.

zookeeper1: Configuration Issues

    * zookeeper1: Service zookeeper1 has 1 Server. Cloudera suggests at least 3 Servers for ZooKeeper.
    * server (masternode): No host heartbeat; CDH versions cannot be verified.


mgmt1: Configuration Issues

    * mgmt1: Time-Series Cache Retention Period
      The recommended heap size that supports a time series cache for a period of 370 minutes is at least 111323720 bytes, only 62801264 are configured.
    * mgmt1: Time-Series Cache Retention Period
      The recommended heap size that supports a time series cache for a period of 370 minutes is at least 111323720 bytes, only 62801264 are configured.
    * servicemonitor (masternode): Time-Series Cache Retention Period
      The recommended heap size that supports a time series cache for a period of 370 minutes is at least 111323720 bytes, only 62801264 are configured.
    * hostmonitor (masternode): Time-Series Cache Retention Period
      The recommended heap size that supports a time series cache for a period of 370 minutes is at least 111323720 bytes, only 62801264 are configured.

Cloudera Employee
Posts: 35
Registered: ‎08-08-2013

Re: Get Cloudera Manager back to good health

The lack of heartbeat explains a lot of the issues. How did you replace your failed node? Is the CM agent running? Can you look at /etc/cloudera-scm-agent/config.ini of the new host? Does it have the right `server_host'?

Highlighted
Explorer
Posts: 13
Registered: ‎10-28-2013

Re: Get Cloudera Manager back to good health

Thanks for your reply . It helps a great deal !!!

 

Actually I have yet to install my two replacement host PC's. This is due soon.

 

By the way I have noticed that my lack of chart facilities could be due to the fact that I do not have Adobe Flash player installed.

 

Also I only temporarily disabled iptables during Cloudera Manager installation. Perhaps part of the reason for the lack of a heartbeat is that these need to be permanently disabled across the cluster.

 

I shall have more info later !!!

 

Francesco

 

 

Explorer
Posts: 10
Registered: ‎07-29-2014

Re: Get Cloudera Manager back to good health

Hi Francesco,

Did you get it up and running ? if yes, would you like to share how ?

Cheers