App Timeline Server not start

[#1]

we cant start the App Timeline Server

we get the following logs when we start the - App Timeline Server

how to fix this isshue ?

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/application_timeline_server.py", line 155, in <module>
    ApplicationTimelineServer().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 314, in execute
    method(env)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 762, in restart
    self.start(env, upgrade_type=upgrade_type)
  File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/application_timeline_server.py", line 44, in start
    self.configure(env) # FOR SECURITY
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 117, in locking_configure
    original_configure(obj, *args, **kw)
  File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/application_timeline_server.py", line 55, in configure
    yarn(name='apptimelineserver')
  File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk
    return fn(*args, **kwargs)
  File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py", line 337, in yarn
    mode=0755
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 555, in action_create_on_execute
    self.action_delayed("create")
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 552, in action_delayed
    self.get_hdfs_resource_executor().action_delayed(action_name, self)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 288, in action_delayed
    self._set_mode(self.target_status)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 459, in _set_mode
    self.util.run_command(self.main_resource.resource.target, 'SETPERMISSION', method='PUT', permission=self.mode, assertable_result=False)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 199, in run_command
    raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of 'curl -sS -L -w '%{http_code}' -X PUT 'http://master01.sys65.com:50070/webhdfs/v1/ats/done?op=SETPERMISSION&user.name=hdfs&permission=755'' returned status_code=403. 
{
  "RemoteException": {
    "exception": "RetriableException", 
    "javaClassName": "org.apache.hadoop.ipc.RetriableException", 
    "message": "org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot set permission for /ats/done. Name node is in safe mode.\nThe reported blocks 425 needs additional 6 blocks to reach the threshold 0.9900 of total blocks 435.\nThe number of live datanodes 3 has reached the minimum number 0. Safe mode will be turned off automatically once the thresholds have been reached."
  }
}

[#2 · Accepted solution]

@Michael Bronson

Your name node is in safe mode you, to confirm do this as root user

# su - hdfs

As hdfs

$ hdfs dfsadmin -safemode get 

From the above you will confirm the status in the error "safe mode" then

$ hdfs dfsadmin -safemode leave 

Then retry

[#3]

@Michael Bronson, HDFS in this cluster is in safemode. Thats why Timelineserver is failing to start. Kindly check HDFS log to see why is Namenode is in safemode. You can explicitly turn off safemode by running "hdfs dfsadmin -safemode leave".