Reply
Explorer
Posts: 23
Registered: ‎12-02-2014

Unable to issue query: the Host Monitor is not running

Hello,

 

I've seen other threads on this issue, but the steps on those threads did not help me solve this issue. Thank you for the help!

 

Specs:

Private cloud

CDH 5.4.2

4 instances

 

Background:

 

The cluster was working fine for ~2 weeks. I noticed last week that the service manager was down, but ignored it as all the services were working fine. Today, I tried working with Kafka and got a java.net.ConnectionRefused exception, then noticed I couldn't start spark-shell as it said it can't connect to node2. 

 

Under "Instances" it says I haven't received a heartbeat from node2 for 13 days, all the other nodes seem to be fine.

 

I can't execute "service cloudera-scm-agent start" on node2, this is the output:

 

/etc/init.d/cloudera-scm-agent: line 123: /var/log/cloudera-scm-agent/cloudera-scm-agent.out: Read-only file system
Starting cloudera-scm-agent: /etc/init.d/cloudera-scm-agent: line 128: /var/run/cloudera-scm-agent.pid: Read-only file system
/etc/init.d/cloudera-scm-agent: line 126: /var/log/cloudera-scm-agent/cloudera-scm-agent.out: Read-only file system
[FAILED]

 

 

I've also just tried restarting the cluster. Stumped on this one :/. Thanks for the help!

Explorer
Posts: 12
Registered: ‎10-30-2013

Re: Unable to issue query: the Host Monitor is not running

First I'd check to make sure /var/log isn't full. Second, that the permissions on it have not changed.

Highlighted
Posts: 1,116
Topics: 1
Kudos: 287
Solutions: 135
Registered: ‎04-22-2014

Re: Unable to issue query: the Host Monitor is not running

Hello

 

"Read-only file system" implies that your disks are not writable.  See if you can touch a file in /var/log and if you get the same error, OS/disk troubleshooting is a good start.

 

- Ben