question Re: Cloudera Manager services not responding after running yum update on CentOS in Support Questions

Cloudera Manager services not responding after running yum update on CentOS

codenchips — Fri, 16 Sep 2022 11:10:06 GMT

Hi I have a problem with my Cloudera cluster which I have not touched for several weeks. It consist of 1 master node and 6 slave nodes(7 VMs). I decided to run yum update and restart on all the nodes. After restarting, I noticed the folowing things:

Symptoms:
- Unable to access Hue at localhost:8888
- Unable to restart cluster via Cloudera Manager(7180)

- Unable to restart Cloudera Management Service
- Cloudera Manager looks like this

Cloudera Manager Screenshot

Steps taken:
- Restart cloudera-scm-server and check status = active(exited)
- Restart cloudera-scm-agent and check status = active(exited)
- Restart cloudera-scm-server-db and check status = server is running

- And various other steps which I forgotten to note down

Please help. I have been troubleshooting this problem for the past day.
Let me know which logs I should attach to provide more information for you guys.

Note: Last resort is to set up everything again. But I have very important codes in my Hue Notebook. Does anyone know the location where Hue Notebooks are stored?

Re: Cloudera Manager services not responding after running yum update on CentOS

Dilshad — Tue, 28 Feb 2017 12:27:03 GMT

did u restart your machine while updating ?

can yo share here logs for cloudera manager ?

Re: Cloudera Manager services not responding after running yum update on CentOS

codenchips — Tue, 28 Feb 2017 12:43:36 GMT

Hi there. Nope. I only restarted my machine after finish running the yum update command.
Can you tell me which logs do you need?
var/lib/cloudera-scm-server/?

var/lib/cloudera-scm-agent/?

var/lib/cloudera-scm-server-db/?

Re: Cloudera Manager services not responding after running yum update on CentOS

Dilshad — Tue, 28 Feb 2017 13:10:15 GMT

all check for any exception while starting
var/lib/cloudera-scm-server/
var/lib/cloudera-scm-agent/
var/lib/cloudera-scm-server-db/

Re: Cloudera Manager services not responding after running yum update on CentOS

saranvisa — Tue, 28 Feb 2017 15:11:57 GMT

@codenchips

Go to Cloudera Manager -> Hosts , check the Host status and understand what kind of issue it shows

Also login as root in linux and run the below command

service ntpd status

service ntpd start

service ntpd status

restart the CM and try again

Re: Cloudera Manager services not responding after running yum update on CentOS

codenchips — Wed, 01 Mar 2017 03:59:22 GMT

Took me some time to copy them online. Here you go, @Dilshad

From Master node:
var/log/cloudera-scm-server/
http://pastebin.com/5AiCP7Wm

var/lib/cloudera-scm-server-db/data/pg_log

http://pastebin.com/L5S71JRx

From Slave node:
var/log/cloudera-scm-server-agent/

http://pastebin.com/k8LMAGG8

Re: Cloudera Manager services not responding after running yum update on CentOS

codenchips — Wed, 01 Mar 2017 04:00:07 GMT

Hi there @saranvisa ! I have checked the Host status. It shows "Unknown Health".
As for your 2nd suggestion, I have checked on all the nodes. service ntpd was not running for some reason(it was running before this) and have restarted them and restarted the CM. There was no difference.

Re: Cloudera Manager services not responding after running yum update on CentOS

saranvisa — Wed, 01 Mar 2017 04:46:50 GMT

@codenchips

Go to CM -> Hosts -> Click on each hosts -> Health history (left down) -> share me the details

Re: Cloudera Manager services not responding after running yum update on CentOS

codenchips — Wed, 01 Mar 2017 05:59:22 GMT

@saranvisa
It shows this message in the Health Historty as shown in the screenshot attached below.

"The Event Server is currently unavailable. View the status of the Event Server"

http://imgur.com/a/8IaSJ

Re: Cloudera Manager services not responding after running yum update on CentOS

codenchips — Wed, 01 Mar 2017 08:16:27 GMT

Hey guys! Great news! Problem finally solved after 1.5 days of troubleshooting!

While digging through the error logs, I saw an error message somewhere in the Cloudera Agent log saying "ValueError: too many values to unpack". Then I search for solutions online to solve that problem.

In conclusion, the errors were caused by
1) System time synchronization was disabled. All nodes's system time was not in sync.
2) Latest Java OpenJDK update broke Cloudera Agent.

Solution:

1) Enabling system time syncronization(As suggested by @saranvisa)

service ntpd start

2) Uninstalling OpenJDK on each node.

rpm -qa | grep jdk

yum remove <each item from the previous step>

3) Run "Re-run upgrade Wizard" in Cloudera Manager and wait for Inspect Host to finish. Done!

Thanks so much for the help guys!

Reference:
https://community.cloudera.com/t5/Cloudera-Manager-Installation/Problem-with-cloudera-agent/td-p/47698/page/2
https://community.cloudera.com/t5/Cloudera-Manager-Installation/Mismatched-CDH-versions-host-has-NONE-but-role-expect-5/m-p/48780#U48780