Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Flume - HDFS HA

SOLVED Go to solution

Flume - HDFS HA

Champion Alumni

Hello,

 

I'm searching for the recommanded configuration for Flume - HDFS Sink when we are using HDFS in HA. 

 

In fact, each time that we restart the cluster/ the nodename fails the active nodename changes and flume fails since is

asking informations on the standby node. 

 

Thank you!

 

Alina

GHERMAN Alina
1 ACCEPTED SOLUTION

Accepted Solutions

Re: Flume - HDFS HA

Master Guru
What form of HDFS path are you configuring in your Flume agent configs?

For HA, you must use the HA service name, such as
hdfs://nameservice1/user/foo instead of
hdfs://namenode-host:8020/user/foo. This will protect your agents from
failures during HA failovers.

3 REPLIES 3

Re: Flume - HDFS HA

Master Guru
What form of HDFS path are you configuring in your Flume agent configs?

For HA, you must use the HA service name, such as
hdfs://nameservice1/user/foo instead of
hdfs://namenode-host:8020/user/foo. This will protect your agents from
failures during HA failovers.

Re: Flume - HDFS HA

Explorer
This is not useful for a remote hdfs clusters... Is possible to user webhdfs from flume?

Re: Flume - HDFS HA

Master Guru
For remote HDFS clusters, just ensure to define the required namespace resolving configuration in your HDFS Gateway hdfs-site.xml configuration. Then in Flume you can use the remote namespace defined name. See http://community.cloudera.com/t5/Storage-Random-Access-HDFS/distcp-with-same-nameservicename/m-p/493... for more details on how to define this.