Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

4 Datanodes started, 3/4 dataNodes are live in ambari UI

4 Datanodes started, 3/4 dataNodes are live in ambari UI

New Contributor

exit

I installed Hadoop cluster with Ambari(HDP2.3). it contains 1 Client node, 2 master node and 4 slaves. First i have 3 slave node with no issue. i added one more slvave and issue started. How can i solve 3/4 DataNodes Live? Please see the attached screenshot.

110298-1565582806363.png

and Datanode Information keep chaging.(data03 and data04)

110299-1565582898852.png

110303-1565582914503.png

hdfs datanode log(user: hdfs)

110323-1565582961560.png

hdfs datanode log(user: root)

110249-1565583020591.png

3 REPLIES 3

Re: 4 Datanodes started, 3/4 dataNodes are live in ambari UI

New Contributor

hdfs datanod log(user:root) is changed.

110304-1565583154749.png

Re: 4 Datanodes started, 3/4 dataNodes are live in ambari UI

Super Mentor

@parkwj93 

We basically see two different errors here in DataNode logs:

 

1.  Following errors indicates that the "dfs.datanode.data.dir" directory (/hadoop/hdfs/data) is not valid ... May be it has some mount issue of the content of the mentioned directory are not readable/corrupt/incorrect permission due to some file system issue.

all directories in dfs.datanode.data.dir are invalid.
directory not readable: /hadoop/hdfs/data

 

2. Port Already is used 50010

BindException: Address already in use 
BindException: Problem binding to [0.0.0.0:50010]


So please check if the directory is valid and readable by the user who is running the DataNode ?
Also before starting the DataNode please try to find and kill a process which is already using port 50010 then freshly try to start the DataNode.

# netstat -tnlpa | grep 50010



.

 

Re: 4 Datanodes started, 3/4 dataNodes are live in ambari UI

Expert Contributor

From the sent screenshots both errors regarding port binding and missing directory seem to originate from the Edge Server (where there should not be any DataNode from your first screenshots).

Can you please double-check that you are not trying to run a DataNode on your Edge Node? The logs may not be relevant then.

 

From your other screenshot there might be a problem in the interconnection of DataNodes and NameNode. Can you please check the NameNode log?

Don't have an account?
Coming from Hortonworks? Activate your account here