Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Where does CDH5 store the list of data nodes?

Solved Go to solution

Where does CDH5 store the list of data nodes?

Rising Star

I have built and existing demo cluster. It was working perfectly well.

 

however because of some changes, I have to change the IP address of each of the data nodes.

 

I did a grep -R 'oldIP' /etc on each machine and edited the files which contained the old IP addresses and replaced them with new IP.

 

I rebooted each machine.

 

However despite doing that when I do

 

sudo -u hdfs hadoop dfsadmin -report

 

it shows me 2 dead data nodes and it lists the old IP addresses.

 

How can I remove old IP and then replace them with new IP addresses?

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Where does CDH5 store the list of data nodes?

Rising Star

this command did not solve the issue. 

 

So I deleted my cluster and rebuilt it using the right IP addresses. 

 

5 REPLIES 5

Re: Where does CDH5 store the list of data nodes?

Rising Star

This is the error. 

 

CRITICAL Initialization failed for Block pool BP-1219478626-192.168.1.20-1418484473049 (Datanode Uuid null) service to nn1home/10.192.128.227:8022 Datanode denied communication with namenode because the host is not in the include-list: DatanodeRegistration(10.192.128.231, datanodeUuid=ff6a2644-3140-4451-a59f-496478a000d7, infoPort=50075, ipcPort=50020, storageInfo=lv=-56;cid=cluster18;nsid=850143528;c=0) at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.registerDatanode(DatanodeManager.java:889) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.registerDatanode(FSNamesystem.java:4798) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.registerDatanode(NameNodeRpcServer.java:1037) at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.registerDatanode(DatanodeProtocolServerSideTranslatorPB.java:92) at org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodeProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:26378) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

Highlighted

Re: Where does CDH5 store the list of data nodes?

Contributor

Are you using cloudera cm? you have to remove the old ip from the hadoop config and add the new ones.

Re: Where does CDH5 store the list of data nodes?

Super Collaborator

hadoop dfsadmin -refreshNodes

 

will help you regenerate the datanode list....

Re: Where does CDH5 store the list of data nodes?

If you use Cloudera Manager, you can look at the list of hosts and try
to recomission any nodes.

If not, you need to look at the file pointed to by the dfs.hosts
property and ensure the new IP address or host name is listed in it.
Then as TGrayson mentioned, you should run sudo -u hdfs hadoop
dfsadmin -refreshNodes"

Let us know if it helped.


Regards,
Gautam Gopalakrishnan

Re: Where does CDH5 store the list of data nodes?

Rising Star

this command did not solve the issue. 

 

So I deleted my cluster and rebuilt it using the right IP addresses. 

 

Don't have an account?
Coming from Hortonworks? Activate your account here