YARN: cannot start App Timeline Server

[#1]

I have one live datanode:
NameNode Started No alertsSNameNode Started No alertsDataNodes1/1 
StartedDataNodes Status1 live / 0 dead / 0 decommissioning

Still i have this exception
File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 250, in _run_command
    raise WebHDFSCallException(err_msg, result_dict)
resource_management.libraries.providers.hdfs_resource.WebHDFSCallException: Execution of 'curl -sS -L -w '%{http_code}' -X PUT 'http://ambari:50070/webhdfs/v1/ats/done?op=SETPERMISSION&user.name=hdfs&permission=755'' returned status_code=403. 
{
  "RemoteException": {
    "exception": "SafeModeException", 
    "javaClassName": "org.apache.hadoop.hdfs.server.namenode.SafeModeException",  

"message": "Cannot set permission for /ats/done. Name node is in safe mode.\nThe reported blocks 9118 needs additional 21 blocks to reach the threshold 1.0000 of total blocks 9138.\nThe number of live datanodes 1 has reached the minimum number 0. Safe mode will be turned off automatically once the thresholds have been reached."

[#2]

happened after I removed "home" from NameNode and DataNode directories

[#3]

@Victor L looks like your namenode has issues -can you restart namenode and then attempt again to see if that helps?

[#4 · Accepted solution]

@Victor L

Looks like your NameNode is started but it is still in SafeMode hence it will not allow any mutable operations to be performed on HDFS. Thats the reason you are getting this SafeMode related error.

In order to validate the same you can try creating a simple directory on HDFS and you might get the same error kind of

# su - hdfs -c "hdfs dfs -mkdir /tmp/test"

If you want to know if the NameNode is in SafeMode or not then you can run the following command:

# su - hdfs -c "hdfs dfsadmin -safemode get"

In order to tell NameNode to leave the safe mode you can run the following command:

# su - hdfs -c "hdfs dfsadmin -safemode leave"

If the NameNode still does not come out of SafeMode then you will need to try restarting the NameNode once and check the NameNode log for any error/warning.

# su -l hdfs -c "/usr/hdp/current/hadoop-hdfs-namenode/../hadoop/sbin/hadoop-daemon.sh stop namenode"
# su -l hdfs -c "/usr/hdp/current/hadoop-hdfs-namenode/../hadoop/sbin/hadoop-daemon.sh start namenode"
# tail -f /var/log/hadoop/hdfs/hadoop-hdfs-namenod*

.

Please share the NN logs if it still does not start cleanly.

.

[#5]

Getting it out of safe mode solved the problem