Flume - HDFS HA

[AlinaGHERMAN]

Hello,

I'm searching for the recommanded configuration for Flume - HDFS Sink when we are using HDFS in HA. 

In fact, each time that we restart the cluster/ the nodename fails the active nodename changes and flume fails since is

asking informations on the standby node. 

Thank you!

Alina

[Harsh J Cloudera Employee · Accepted solution]

What form of HDFS path are you configuring in your Flume agent configs?

For HA, you must use the HA service name, such as
hdfs://nameservice1/user/foo instead of
hdfs://namenode-host:8020/user/foo. This will protect your agents from
failures during HA failovers.

[marccasajus]

This is not useful for a remote hdfs clusters... Is possible to user webhdfs from flume?

[Harsh J Cloudera Employee]

For remote HDFS clusters, just ensure to define the required namespace resolving configuration in your HDFS Gateway hdfs-site.xml configuration. Then in Flume you can use the remote namespace defined name. See http://community.cloudera.com/t5/Storage-Random-Access-HDFS/distcp-with-same-nameservicename/m-p/493... for more details on how to define this.