Skip to main content
Manoj690
Manoj690
Silver ⭐️⭐️⭐️
August 21, 2019
Solved

In ambari Namenode not started

this is the error

 

2019-08-21 16:13:10,828 - The NameNode is still in Safemode. Please be careful with commands that need Safemode OFF. Traceback (most recent call last): File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HDFS/package/scripts/namenode.py", line 408, in <module> NameNode().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute method(env) File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HDFS/package/scripts/namenode.py", line 138, in start upgrade_suspended=params.upgrade_suspended, env=env) File "/usr/lib/ambari-agent/lib/ambari_commons/os_family_impl.py", line 89, in thunk return fn(*args, **kwargs) File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HDFS/package/scripts/hdfs_namenode.py", line 264, in namenode create_hdfs_directories(name_service) File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HDFS/package/scripts/hdfs_namenode.py", line 336, in create_hdfs_directories nameservices=name_services File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__ self.env.run() File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run self.run_action(resource, action) File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action provider_action() File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 677, in action_create_on_execute self.action_delayed("create") File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 674, in action_delayed self.get_hdfs_resource_executor().action_delayed(action_name, self) File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 373, in action_delayed self.action_delayed_for_nameservice(None, action_name, main_resource) File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 395, in action_delayed_for_nameservice self._assert_valid() File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 334, in _assert_valid self.target_status = self._get_file_status(target) File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 497, in _get_file_status list_status = self.util.run_command(target, 'GETFILESTATUS', method='GET', ignore_status_codes=['404'], assertable_result=False) File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 214, in run_command return self._run_command(*args, **kwargs) File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 282, in _run_command _, out, err = get_user_call_output(cmd, user=self.run_user, logoutput=self.logoutput, quiet=False) File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output raise ExecutionFailed(err_msg, code, files_output[0], files_output[1]) resource_management.core.exceptions.ExecutionFailed: Execution of 'curl -sS -L -w '%{http_code}' -X GET -d '' -H 'Content-Length: 0' 'http://gaian-lap386.com:50070/webhdfs/v1/tmp?op=GETFILESTATUS&user.name=hdfs' 1>/tmp/tmpPLc7iw 2>/tmp/tmpCp00V8' returned 7. curl: (7) Failed to connect to gaian-lap386.com port 50070: Connection refused 000 

 

After i use this safemode leave command below result i got

 

hdfs dfsadmin -safemode leave
safemode: Call From gaian-lap386.com/192.168.24.32 to gaian-lap386.com:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused

jsensharma
jsensharmaAnswer
Platinum ⭐️⭐️
August 21, 2019

@Manoj690 

Not the same output as mine. Please notice that,  In your case that file is being owned by "root" user.  Not the "hdfs" user. 

So please try this:

 

 

# chown hdfs:hadoop /hadoop/hdfs/namenode/current/VERSION

(OR recursively)
# chown -R hdfs:hadoop /hadoop/hdfs/

 

 

 

View in thread

7 replies

jsensharma
Platinum ⭐️⭐️
jsensharma
Platinum ⭐️⭐️
August 21, 2019

@Manoj690 

Looks like the NameNode is not starting successfully.  Can you please check and share the NameNode logs.

 

It looks like a similar thread is also opened by you here:   https://community.cloudera.com/t5/Support-Questions/In-ambari-all-my-servies-are-down/m-p/268512/highlight/false#M206243

Please use the same troubleshooting steps to collect the data for troubleshooting.

For example

Please check if the NameNode is running or not? 

# ps -ef | grep -i NameNode


If NameNode port is listening on 50070? 

# netstat -tnlpa | grep 50070

 

If not then please check and share the NameNode log. If possible then try to attach the Full NameNode logs.



Translate
Manoj690
Manoj690Author
Silver ⭐️⭐️⭐️
August 21, 2019

As i mentioned  name node is in safemode. 

 

ps -ef | grep -i NameNode
root 30876 5714 0 17:21 pts/3 00:00:00 grep --color=auto -i NameNode

 

netstat -tnlpa | grep 50070

 

and

how to check namenode log files 

Translate
jsensharma
Platinum ⭐️⭐️
jsensharma
Platinum ⭐️⭐️
August 21, 2019

@Manoj690 
We see the error as following:

java.io.FileNotFoundException: /hadoop/hdfs/namenode/current/VERSION (Permission denied)

 So that is causing the NameNode startup failure.

Please check and fix the permission on the mentioned file so that the Owner of the NameNode process will have read / write access to this file.

 

Normally it is owned by hdfs user.  

Example:

# ls -l /hadoop/hdfs/namenode/current/VERSION
-rw-r--r--. 1 hdfs hadoop 206 Aug 13 23:10 /hadoop/hdfs/namenode/current/VERSION

 

Translate
Terms & ConditionsAccessibility statement