11-02-2017 09:35 AM - last edited on 11-21-2017 05:25 AM by cjervis
I couldn't figure out what went wrong with our Cloudera cluster, there are multiple errors but most important one is that I can't access Cloudera Manager web UI at https://<cmserverip>:7183/cmf/home. It throws ERR_CONNECTION_CLOSED on Chrome.
If I run netstat -anp | grep 7183, it's 5000+ lines of CLOSED_WAITs with a bunch of ESTABLISHEDs all from the localhost IP
* Selinux is disabled. Iptables are empty.
* hostname -f returns correct name.
* /etc/hosts file are all same and correct on all nodes
* I also can't access any of the web UIs of the Hadoop services (such as Hue) on the cluster. They throw ERR_CONNECTION_TIMED_OUT on Chrome.
* All CDH services are up and accessible through command-line.
* None of logs I did find anything unusual.
* I CAN access non-CDH services on the nodes like RStudio or Jupyter.
I'm suspecting something failed with certification stuff or Kerberos but I don't know how to check those things. Would appreciate any help.
11-20-2017 11:28 PM
SOLVED: The problem was the CM's mysql database. Navigator index data filled up the disk partition that mysql data is in. Once I cleaned it CDH services worked.