@sparkd,   While we can't be sure, it is likely that some permissions were changed on the /tmp directory so that the Service Monitor (that executes the HDFS canary health check) could not access the directory.  Service Monitor utilizes the "hue" user and principal to access other resources so it is reasonable to assume that /tmp in HDFS did not allow the hue user or group to write to /tmp.   Are you having similar trouble?  If so, check your service monitor log file for stack traces and errors related to the hdfs canary. ... View more

it could be that the file is created (locked) by other user than cloudera-scm - have you attempted to delete and restart the cloudera-scm-server-db ? ... View more