Ambari errors out with 500 status code

[#1]

SYMPTOM

Sometimes while performing any Ambari operation like for example, adding a Ranger KMS service or doing any other operation, the Ambari UI (or even curt response) might show this error:

ERROR:

Error 500 status code received on GET method for API: /api/v1/stacks/HDP/versions/2.3/recommendations
Error message: Error occurred during stack advisor command invocation: Cannot create /var/run/ambari-server/stack-recommendations

ROOT CAUSE This is most likely a permission issue with /var/run/ambari-server/ location for the user who is running the 'ambari-server' process.

RESOLUTION To resolve this, the permission for /var/run/ambari/server should be setup correctly. On the good cluster, where the ambari-server is running as 'root' the permission looks like:

[root@test ~]# ll -d /var/run/ambari-server/
drwxr-xr-x 4 root root 4096 Dec  4 05:10 /var/run/ambari-server/


[root@test ~]# ll /var/run/ambari-server/
total 12
-rw-r--r--  1 root root  6 Dec  4 05:10 ambari-server.pid
drwxr-xr-x  4 root root 4096 Dec  4 05:26 bootstrap
drwxr-xr-x 39 root root 4096 Feb 17 18:14 stack-recommendations

Fix the permissions and try the operations on Ambari dashboard.