HDFS Name node high avaliability

[#1]

Hi ,

I have tested Cloudera 5.11 High avaliability active passive with 2 name nodes and noticed that when name node or standby node machines are down the HDFS is not available.

when stoping name node on node 4 getting the error :

hadoop fs -ls /user/hive/warehouse/dwh.db/

llegalArgumentException: java.net.UnknownHostException: gc-test-impala28-04.c.bi-environment-1271.internal error 400

Do I miss anything ?

Thanks

Alon

Spoiler

[#2]

How did you set up NN AH? Can you provide the context of the two NN, their names, which one you took down so that make it a more clear on what is going wrong here?

The error itself is complaining that the host cannot be resolved. This could be a config issue with the NN HA setup as there should be a service name that resolves, through the config files, to both host. So when NN1 goes offline it will try the NN2. This could be a host name resolution error as well. If DNS or hosts files are not set up correctly then it is simply not able to resolve NN2 from the client.

[#3]

Hi ,

Thanks for the quick reply .

I used the following instructions :

https://www.cloudera.com/documentation/enterprise/5-8-x/topics/cdh_hag_hdfs_ha_enabling.html#topic_2...

The service name is nameservice1 (the default) combined from node 4 (active) and node 3(standby)

as I mentioned when shutting down node 4 it seems that the name service is still looking for node 4 .

otherwise it not clear why I'm getting this error .

If it was possible I would add screen shots to the post .

node 3 & 4 roles are :

datanode

failover controler

journal node

name node

zookeepr

node manager

hive gateway

The cloudera version is 5.11

Thanks

Alon

[#4]

If it is still looking for node4 after it goes down then the standby is not in a valid state, could be a quorum issue between the journal nodes, or a state/fencing problem with ZK. Review the logs for these services during the test to try to glean more information.

[#5]

Thanks 🙂

Checked the logs got the message :

java.net.ConnectException: Call From gc-test-impala28-03.c.bi-environment-1271.internal/10.128.0.46 to gc-test-impala28-03.c.bi-environment-1271.internal:8022 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: "http://wiki.apache.org/hadoop/ConnectionRefused"

it seems that for some reason it could not connect to the second name node port 8022 .

Checked if port is listening and got :

root@gc-test-impala28-03:~# netstat -tulpan | grep 8022
tcp        0      1 10.128.0.46:44608       10.128.0.45:8022        SYN_SENT    16026/java

Any Idea ?

Thanks

Alon