Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

All services are in a BAD health after Cloudera installation

All services are in a BAD health after Cloudera installation

New Contributor

Hello All.

 

I have just installed CDH 5.4.2 on RHEL 6.6 on AWS EC2. Very simple setup - initially just 1 node with Core Hadoop services. 

Installation was successful, no error messages.

When I logged into Cloudera Manager, it showed ALL services in a BAD health. Health issues are the following:

Cluster 1

 

HDFS
Under-Replicated Blocks, NameNode Health

 

YARN (MR2 Included)
ResourceManager Health

 

ip-172-31-12-101.us-west-2.compute.internal

 Clock Offset

 

Cloudera Management Services

 

eventserver (ip-172-31-12-101)
Unexpected Exits, Web Server Status

 

navigator (ip-172-31-12-101)
Web Server Status, Unexpected Exits

 

reportsmanager (ip-172-31-12-101)
Unexpected Exits, Process Status

 

I tried to:
- Restart Cloudera Management Service
- Reboot the Linux server
- Adjusted Clock Offset Threshold on Host to a very big value.

So far, nothing helped.

 

Therefore, I have 2 questions:

1. Is it a normal situation that Cloudera works so weird after successful setup or did I do anything wrong?
2. How to resolve these BAD health issues?

 

Will appreciate your help.

 

Thanks in advance.

4 REPLIES 4

Re: All services are in a BAD health after Cloudera installation

New Contributor
Quick update. After I completely disabled Host Clock Offset Thresholds, all services, except HDFS and Cloudera Management Service related ones turned into good health.

Really weird behavior provided that I don't use NTP server...

Re: All services are in a BAD health after Cloudera installation

NTP is highly recommended. It is also possible that the hosts you are using are a bit underpowered. What instance type is in use?
Regards,
Gautam Gopalakrishnan

Re: All services are in a BAD health after Cloudera installation

New Contributor
Thanks, but I already resolved the issues. The last one with under replicated blocks was caused by replication factor 3 while I had 2 HDFS data nodes at that time. When I added additional data nodes, the issue got resolved.

Re: All services are in a BAD health after Cloudera installation

New Contributor

Hi Gautam,

I am in exactly same situation where Installation, health check was successful initially but went bad just in couple of days without my changing anything significantly.

 

I mostly get this Clock Offset error alterts almost daily for most all (definitely YARN and HDFS) services.

 

FYI installed 5.3.1 in RHEL 6.5 4 node cluster somewhere by Feb/March and don't know top of my head whether I installed/configured NTP service. Would NTP be recommended for 5.3 as well because I saw online doc that is for 5.4.

 

Could you please comment?

Thanks,

Paddy