Skip to main content
Manoj690
Manoj690
Silver ⭐️⭐️⭐️
August 21, 2019
Solved

In ambari Namenode not started

this is the error

 

2019-08-21 16:13:10,828 - The NameNode is still in Safemode. Please be careful with commands that need Safemode OFF. Traceback (most recent call last): File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HDFS/package/scripts/namenode.py", line 408, in <module> NameNode().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute method(env) File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HDFS/package/scripts/namenode.py", line 138, in start upgrade_suspended=params.upgrade_suspended, env=env) File "/usr/lib/ambari-agent/lib/ambari_commons/os_family_impl.py", line 89, in thunk return fn(*args, **kwargs) File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HDFS/package/scripts/hdfs_namenode.py", line 264, in namenode create_hdfs_directories(name_service) File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HDFS/package/scripts/hdfs_namenode.py", line 336, in create_hdfs_directories nameservices=name_services File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__ self.env.run() File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run self.run_action(resource, action) File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action provider_action() File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 677, in action_create_on_execute self.action_delayed("create") File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 674, in action_delayed self.get_hdfs_resource_executor().action_delayed(action_name, self) File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 373, in action_delayed self.action_delayed_for_nameservice(None, action_name, main_resource) File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 395, in action_delayed_for_nameservice self._assert_valid() File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 334, in _assert_valid self.target_status = self._get_file_status(target) File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 497, in _get_file_status list_status = self.util.run_command(target, 'GETFILESTATUS', method='GET', ignore_status_codes=['404'], assertable_result=False) File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 214, in run_command return self._run_command(*args, **kwargs) File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 282, in _run_command _, out, err = get_user_call_output(cmd, user=self.run_user, logoutput=self.logoutput, quiet=False) File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output raise ExecutionFailed(err_msg, code, files_output[0], files_output[1]) resource_management.core.exceptions.ExecutionFailed: Execution of 'curl -sS -L -w '%{http_code}' -X GET -d '' -H 'Content-Length: 0' 'http://gaian-lap386.com:50070/webhdfs/v1/tmp?op=GETFILESTATUS&user.name=hdfs' 1>/tmp/tmpPLc7iw 2>/tmp/tmpCp00V8' returned 7. curl: (7) Failed to connect to gaian-lap386.com port 50070: Connection refused 000 

 

After i use this safemode leave command below result i got

 

hdfs dfsadmin -safemode leave
safemode: Call From gaian-lap386.com/192.168.24.32 to gaian-lap386.com:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused

jsensharma
jsensharmaAnswer
Platinum ⭐️⭐️
August 21, 2019

@Manoj690 

Not the same output as mine. Please notice that,  In your case that file is being owned by "root" user.  Not the "hdfs" user. 

So please try this:

 

 

# chown hdfs:hadoop /hadoop/hdfs/namenode/current/VERSION

(OR recursively)
# chown -R hdfs:hadoop /hadoop/hdfs/

 

 

 

View in thread

7 replies

jsensharma
Platinum ⭐️⭐️
jsensharma
Platinum ⭐️⭐️
August 21, 2019

@Manoj690 

Looks like the NameNode is not starting successfully.  Can you please check and share the NameNode logs.

 

It looks like a similar thread is also opened by you here:   https://community.cloudera.com/t5/Support-Questions/In-ambari-all-my-servies-are-down/m-p/268512/highlight/false#M206243

Please use the same troubleshooting steps to collect the data for troubleshooting.

For example

Please check if the NameNode is running or not? 

# ps -ef | grep -i NameNode


If NameNode port is listening on 50070? 

# netstat -tnlpa | grep 50070

 

If not then please check and share the NameNode log. If possible then try to attach the Full NameNode logs.



Translate
Manoj690
Manoj690Author
Silver ⭐️⭐️⭐️
August 21, 2019

As i mentioned  name node is in safemode. 

 

ps -ef | grep -i NameNode
root 30876 5714 0 17:21 pts/3 00:00:00 grep --color=auto -i NameNode

 

netstat -tnlpa | grep 50070

 

and

how to check namenode log files 

Translate
jsensharma
Platinum ⭐️⭐️
jsensharma
Platinum ⭐️⭐️
August 21, 2019

@Manoj690 

You can find NameNode logs here:

# less /var/log/hadoop/hdfs/hadoop-hdfs-namenode-gaian-lap386.com.log
# less /var/log/hadoop/hdfs/hadoop-hdfs-namenode-gaian-lap386.com.out

 

Similarly you can find the DataNode logs somewhere "/var/log/hadoop/hdfs/hadoop-hdfs-datanode-xxxxxxx.log"

Can you please attach those logs here. Or please share the erros from the mentioned files.   (preferred is to attach the files here)

Translate
Terms & ConditionsAccessibility statement