Ambari Alert with Operation not permitted: '/var/lib/ambari-agent/tmp/curl_krb_cache'

[#1]

PROBLEM: Ambari alert with Operation not permitted: '/var/lib/ambari-agent/tmp/curl_krb_cache' error

Alert:

ERROR 2017-04-27 20:17:10,287 alert_ha_namenode_health.py:185 - [Alert] NameNode High Availability Health on taco1.example.com fails:
Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/alerts/alert_ha_namenode_health.py", line 171, in execute
    kinit_timer_ms = kinit_timer_ms)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/curl_krb_request.py", line 106, in curl_krb_request
    os.chmod(curl_krb_cache_path, 0777)
OSError: [Errno 1] Operation not permitted: '/var/lib/ambari-agent/tmp/curl_krb_cache'

ROOT CAUSE: Ambari running as non-root user and /var/lib/ambari-agent is owned by root.

SOLUTION: Move the directory to tmp location or change the ownership of the directory per non-root user.

$ mv /var/lib/ambari-agent/tmp /var/lib/ambari-agent/tmp.old

OR

$ chown <non-root-user>:<non-root-user> /var/lib/ambari-agent -R