question Re: ViewFS use in java application in Archives of Support Questions (Read Only)

ViewFS use in java application

viruvekariya — Tue, 15 Nov 2016 18:53:00 GMT

Hi,

I have already tested my HDFS HA cluster with my java application (i.e simple file upload/download).

Now that I have configured federated cluster, How can I use it in same code with few necessary modifications. Here is my code example for HDFS HA cluster. Please suggest for Federation HA cluster. Problem is I think my Configuration object is getting null while getting the FileSystem object for federated cluster.

conf.set("fs.defaultFS", "hdfs://HadoopTestHA");

conf.set("dfs.replication","4");
conf.set("dfs.ha.namenodes.HadoopTestHA", "nn1,nn2");
conf.set("dfs.namenode.rpc-address.HadoopTestHA.nn1","hadoop4ind.india:8020");
conf.set("dfs.namenode.rpc-address.HadoopTestHA.nn2","hadoop5ind.india:8020");
conf.set("dfs.client.failover.proxy.provider.HadoopTestHA","org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider");
copyFromLocalHDFS(conf,"Fed_Java_Test");  // File uploading code

Re: ViewFS use in java application

rajkumar_singh — Tue, 27 Dec 2016 00:22:22 GMT

@Viraj Vekaria could you please try setting these properties in conf and see if it helps?

dfs.nameservices=HadoopTestHA

dfs.ha.namenodes.HadoopTestHA=nn1,nn2

dfs.namenode.rpc-address.HadoopTestHA.namenode1=hadoop4ind.india:8020

dfs.namenode.rpc-address.HadoopTestHA.namenode2=hadoop5ind.india:8020

dfs.client.failover.proxy.provider.HadoopTestHA=org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider

Re: ViewFS use in java application

cstanca — Tue, 27 Dec 2016 03:57:27 GMT

In your code, you may have to add the following line:

conf.set("dfs.nameservices", "HadoopTestHA")