Support Questions
Find answers, ask questions, and share your expertise

The nameservice for a Hadoop Namenode HA should be discoverable across clusters.

The nameservice for a Hadoop Namenode HA should be discoverable across clusters.

New Contributor

Requirement: The nameservice for a Hadoop Namenode HA should be discoverable across clusters.

Solution #1: One solution I found online is to add the nameservice configurations to all the hdfs-site.xml files in the clusters involved.

Problem: We have 10 clusters and growing, we cannot add new cluster definitions in all the clusters everytime a new cluster is deployed.

Pros: Manageble for few clusters. Cons: Not scalable to more clusters.

Solution #2: We are planning working on a second solution to have a central service to resolve the name service across clusters and have a custom class extending org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider class.

Please provide any inputs on any better solutions or any existing patches that are already filed for this issue.

2 REPLIES 2
Highlighted

Re: The nameservice for a Hadoop Namenode HA should be discoverable across clusters.

@Mahipal Jupali - Is there any possibility of getting this information via the Ambari API? Sorry I can't give a full answer, but maybe this would be a suitable starting point: https://cwiki.apache.org/confluence/display/AMBARI/Modify+configurations

Highlighted

Re: The nameservice for a Hadoop Namenode HA should be discoverable across clusters.

New Contributor

@Justin Watkins - Thanks for your response. The problem is not propagating the changes in the cluster, we have to configure hdfs-site.xml and restart all the services every time we add a new cluster, which is not feasible.