Impala Daemon Not Starting - "Process" Directory Not Created

[#1]

One of our Impala daemons will not start.  There is no directory structure being created at:

/var/run/cloudera-scm-agent/process/nnnnnn-impala-IMPALAD

After the daemon fails to start, the stderr and stdout logs cannot be accessed because the directory structure does not exist.  Instead, there is an HTTP 403 error, referencing the expected path, indicating "The server declined access to the page or resource."  In addition, nothing is being written to the error, warning, or info logs at /hadoop/log/impalad.

What can be done to enable the creation of the directory structure?  Deleting and reinstalling the Impala role does not resolve it.  There are no issues with other roles on the host.  I am wondering whether it is necessary to re-initialize the host and re-install the roles.

Michael Reynolds

[#2 · Accepted solution]

When using Cloudera Manager to start the daemons, Cloudera Manager creates a command to be run and tells the host on which the service will start to hearbeat into Cloudera Manager.  The agent heartbeat response from Cloudera Manager will tell the agent that a service needs to start.

It is up to the agent, then to start the process.  If you see no /var/run/... directory being created, that means that something likely went wrong before that was supposed to occur.  Some places to check

Cloudera Manager.  If the command failed to be processed due to a timeout you should see an error in the Command itself.  What do you see there?

Cloudera Manager log (/var/log/cloudera-scm-server.log ) What happens in the log when you try to start the daemon?

Agent log (/var/log/cloudera-scm-agent).  On the host where the Impala daemon should be running, check the agent log for clues.  Did the agent even know that it should start the service?  Are there errors or excpetions during hearbeat attempts?

That's a good place to start.

[#3]

Thank you very much for your reply.  According to the agent log, the service account responsible for running the Impala service does not exist on the host.  I appreciate your pointing me in the right direction.