Reply
Explorer
Posts: 23
Registered: ‎04-03-2017

HDFS Name node high avaliability

Hi ,

 

I have tested Cloudera 5.11 High avaliability active passive with 2 name nodes and noticed that when name node or standby node machines are down the HDFS is not available.

 

when stoping name node on node 4 getting the error :

hadoop fs -ls /user/hive/warehouse/dwh.db/

llegalArgumentException: java.net.UnknownHostException: gc-test-impala28-04.c.bi-environment-1271.internal error 400

 

Do I miss anything ?

 

Thanks

 

Alon

Spoiler
 

 

 

Posts: 642
Topics: 3
Kudos: 103
Solutions: 67
Registered: ‎08-16-2016

Re: HDFS Name node high avaliability

How did you set up NN AH? Can you provide the context of the two NN, their names, which one you took down so that make it a more clear on what is going wrong here?

The error itself is complaining that the host cannot be resolved. This could be a config issue with the NN HA setup as there should be a service name that resolves, through the config files, to both host. So when NN1 goes offline it will try the NN2. This could be a host name resolution error as well. If DNS or hosts files are not set up correctly then it is simply not able to resolve NN2 from the client.
Explorer
Posts: 23
Registered: ‎04-03-2017

Re: HDFS Name node high avaliability

Hi ,

Thanks for the quick reply .

 

I used the following instructions :

 

https://www.cloudera.com/documentation/enterprise/5-8-x/topics/cdh_hag_hdfs_ha_enabling.html#topic_2...

 

The service name is nameservice1 (the default) combined from node 4 (active) and node 3(standby)

 

as I mentioned when shutting down node 4 it seems that the name service is still looking for node 4 .

otherwise it not clear why I'm getting this error .

 

If it was possible I would add screen shots to the post .

 

node 3 & 4 roles are :

datanode

failover controler

journal node

name node

zookeepr

node manager

hive gateway

 

 

The cloudera version is 5.11

 

Thanks

 

Alon

 

 

Posts: 642
Topics: 3
Kudos: 103
Solutions: 67
Registered: ‎08-16-2016

Re: HDFS Name node high avaliability

If it is still looking for node4 after it goes down then the standby is not in a valid state, could be a quorum issue between the journal nodes, or a state/fencing problem with ZK. Review the logs for these services during the test to try to glean more information.
Highlighted
Explorer
Posts: 23
Registered: ‎04-03-2017

Re: HDFS Name node high avaliability

Thanks :-)

 

Checked the logs got the message :

 

java.net.ConnectException: Call From gc-test-impala28-03.c.bi-environment-1271.internal/10.128.0.46 to gc-test-impala28-03.c.bi-environment-1271.internal:8022 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: "http://wiki.apache.org/hadoop/ConnectionRefused"

 

it seems that for some reason it could not connect to the second name node port 8022 .

 

Checked if port is listening and got :

 

root@gc-test-impala28-03:~# netstat -tulpan | grep 8022
tcp        0      1 10.128.0.46:44608       10.128.0.45:8022        SYN_SENT    16026/java

 

Any Idea ?

 

Thanks

 

Alon

Announcements