HA support by using HttpFs with Hue doesn't work

[#1]

Hello,

I am using hue and I want to connect to a secure HDFS cluster. I have 3 Namenodes configured and I am running one httpfs server on each Namenode.I configured hue.ini to use all of them. I can access hdfs cluster from hue ui. However if I kill one of the httpfs server on one of the Namenode where hue initially connected to , hue is not able to switch to another httpfs server on another namenode.

Here is my hue.ini config :

[[hdfs_clusters]]
[[[default]]]
fs_defaultfs=hdfs://crd-dev-sor1p

logical_name=nn1

webhdfs_url=http://blitzhbase02-mnds3-1-crd.eng.sfdc.net:14000/webhdfs/v1

security_enabled=true

hadoop_conf_dir=/home/sfdc/current/bigdata-hadoop/hadoop/hadoop/etc/hadoop

[[[ha]]]
fs_defaultfs=hdfs://crd-dev-sor1p
webhdfs_url=http://blitzhbase02-mnds4-1-crd.eng.sfdc.net:14000/webhdfs/v1
security_enabled=true

[[[ha1]]]
fs_defaultfs=hdfs://crd-dev-sor1p
webhdfs_url=http://blitzhbase02-dnds2-2-crd.eng.sfdc.net:14000/webhdfs/v1
security_enabled=true

[#2 · Accepted solution]

This is expected, the current design for HttpFs HA comes with using a
balancer in front of the HttpFs servers.

Only for the YARN Resource Manager Hue is operating a failover itself.

[#3]

Hi Romain,

Thanks for the reply.

Yes I could setup hue to do failover itself for yarn RM.

HttpFS for hue is already acting as a balancer for Namenodes , now setting up balancer again for HttpFS is like setting up balancer for balancer.

Any guide on setting up balancer before Httpfs for hue and corresponding sample hue.ini for that setup ?

[#4]

Maybe it could be better to launch httpfs on the node with the Hue? In such deployment hue+httpfs node will communicate with the HDFS datanodes directly and not through other server (the Hue machine itself will be the HTTP proxy).

But I guess balancing of the multiple httpfs proxies around the cluster would be usefull even in this case - for high available WebHDFS interface for end users.