HDF 2.1.4 not starting up after reboot

[#1]

My HDF cluster is not starting after a server reboot. I was trying to mimic a recent Data Centre crash due to power failure and implementing the auto reboot of our dev cluster using ambari blueprints. I have errors with the NIFI_CA and NiFi is not starting up. Please see the error message below; Not sure why there is a permission issue...

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi_ca.py", line 114, in <module>
    CertificateAuthority().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi_ca.py", line 92, in start
    Execute((run_ca_script, params.jdk64_home, ca_server_script, params.nifi_config_dir + '/nifi-certificate-authority.json', params.nifi_ca_log_file_stdout, params.nifi_ca_log_file_stderr, status_params.nifi_ca_pid_file), user=params.nifi_user)
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 273, in action_run
    tries=self.resource.tries, try_sleep=self.resource.try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
    result = function(command, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
    tries=tries, try_sleep=try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 293, in _call
    raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of '/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/run_ca.sh /usr/java/jdk1.8.0_77 /var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/files/nifi-toolkit-1.1.0.2.1.4.0-5/bin/tls-toolkit.sh /usr/hdf/current/nifi/conf/nifi-certificate-authority.json /var/log/nifi/nifi-ca.stdout /var/log/nifi/nifi-ca.stderr /var/run/nifi/nifi-ca.pid' returned 126. -bash: /var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/run_ca.sh: Permission denied

[#2 · Accepted solution]

@Joshua Adeleke

The error says :

 -bash: /var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/run_ca.sh: Permission denied

.

Can you please check if you have that file existing and if it has the correct permissions? Can you please check and share the permission of this file?

Are you starting ambari-agent as "root" user or non-root user?

You also might want to clear the ambari-agent "cache" directory and then try restarting the agent so that it can pull fresh files from ambari server.

.

[#3]

@Jay SenSharma

The file has the correct permissions (see below). I am running ambari-agent as "root" user.

-rwxr-xr-x 1 root root 2002 Aug  1 10:44 /var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/run_ca.sh

[#4]

@Jay SenSharma Restarting ambari-server did it for me. Thanks