Support Questions
Find answers, ask questions, and share your expertise

how to reference an HA hadoop cluster using the nameserviceId?

New Contributor

I have an HA hadoop cluster with 2 namenodes and the nameserviceId as "nameservice1" . When I am trying to make connection to this cluster using url like below it gives me error :

 

curl -i "http://nameservice1/webhdfs/v1/user/hdfs/?blocksize=67108864&replication=3&permission=744&dfs.nameservices=nameservice1&dfs.ha.namenodes.nameservice1=master1,master2&dfs.namenode.http-address.nameservice1.master1=master1.hadoop.com:50070&dfs.namenode.http-address.nameservice1.master2=master2.hadoop.com:50070&dfs.namenode.rpc-address.nameservice1.master1=master1.hadoop.com:8022&dfs.namenode.rpc-address.nameservice1.master2=master2.hadoop.com:8022&dfs.defaultFS=hdfs://nameservice1&user=hdfs&op=LISTSTATUS"

 

Ho can one call an HA hadoop cluster using webhdfs?

1 REPLY 1

Re: how to reference an HA hadoop cluster using the nameserviceId?

Super Mentor

@bigdataC 

 

WebHDFS does not natively support Namenode HA (means you can not use "namesrvice" name in the URL) but you can use Knox to provide the functionality.

Like https://cwiki.apache.org/confluence/display/KNOX/Examples+WebHDFS

.
https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.5/configuring-proxy-knox/content/ha_configure_we...

 

SO either you can configure components like Knox in front of nameNodes  Or you will have to use the webhdfs with individual NameNode address:port in the webhdfs URL.