Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here. Want to know more about what has changed? Check out the Community News blog.

All services are in a BAD health after Cloudera installation

All services are in a BAD health after Cloudera installation

New Contributor

Hello All.

 

I have just installed CDH 5.4.2 on RHEL 6.6 on AWS EC2. Very simple setup - initially just 1 node with Core Hadoop services. 

Installation was successful, no error messages.

When I logged into Cloudera Manager, it showed ALL services in a BAD health. Health issues are the following:

Cluster 1

 

HDFS
Under-Replicated Blocks, NameNode Health

 

YARN (MR2 Included)
ResourceManager Health

 

ip-172-31-12-101.us-west-2.compute.internal

 Clock Offset

 

Cloudera Management Services

 

eventserver (ip-172-31-12-101)
Unexpected Exits, Web Server Status

 

navigator (ip-172-31-12-101)
Web Server Status, Unexpected Exits

 

reportsmanager (ip-172-31-12-101)
Unexpected Exits, Process Status

 

I tried to:
- Restart Cloudera Management Service
- Reboot the Linux server
- Adjusted Clock Offset Threshold on Host to a very big value.

So far, nothing helped.

 

Therefore, I have 2 questions:

1. Is it a normal situation that Cloudera works so weird after successful setup or did I do anything wrong?
2. How to resolve these BAD health issues?

 

Will appreciate your help.

 

Thanks in advance.

4 REPLIES 4

Re: All services are in a BAD health after Cloudera installation

New Contributor
Quick update. After I completely disabled Host Clock Offset Thresholds, all services, except HDFS and Cloudera Management Service related ones turned into good health.

Really weird behavior provided that I don't use NTP server...

Re: All services are in a BAD health after Cloudera installation

NTP is highly recommended. It is also possible that the hosts you are using are a bit underpowered. What instance type is in use?
Regards,
Gautam Gopalakrishnan

Re: All services are in a BAD health after Cloudera installation

New Contributor
Thanks, but I already resolved the issues. The last one with under replicated blocks was caused by replication factor 3 while I had 2 HDFS data nodes at that time. When I added additional data nodes, the issue got resolved.

Re: All services are in a BAD health after Cloudera installation

New Contributor

Hi Gautam,

I am in exactly same situation where Installation, health check was successful initially but went bad just in couple of days without my changing anything significantly.

 

I mostly get this Clock Offset error alterts almost daily for most all (definitely YARN and HDFS) services.

 

FYI installed 5.3.1 in RHEL 6.5 4 node cluster somewhere by Feb/March and don't know top of my head whether I installed/configured NTP service. Would NTP be recommended for 5.3 as well because I saw online doc that is for 5.4.

 

Could you please comment?

Thanks,

Paddy