question Re: Flume - HDFS HA in Archives of Support Questions (Read Only)

Flume - HDFS HA

AlinaGHERMAN — Fri, 16 Sep 2022 09:32:52 GMT

Hello,

I'm searching for the recommanded configuration for Flume - HDFS Sink when we are using HDFS in HA.

In fact, each time that we restart the cluster/ the nodename fails the active nodename changes and flume fails since is

asking informations on the standby node.

Thank you!

Alina

Re: Flume - HDFS HA

Harsh J — Wed, 01 Jul 2015 07:28:55 GMT

What form of HDFS path are you configuring in your Flume agent configs?

For HA, you must use the HA service name, such as
hdfs://nameservice1/user/foo instead of
hdfs://namenode-host:8020/user/foo. This will protect your agents from
failures during HA failovers.

Re: Flume - HDFS HA

marccasajus — Mon, 16 Oct 2017 08:54:49 GMT

This is not useful for a remote hdfs clusters... Is possible to user webhdfs from flume?

Re: Flume - HDFS HA

Harsh J — Mon, 16 Oct 2017 09:20:25 GMT

For remote HDFS clusters, just ensure to define the required namespace resolving configuration in your HDFS Gateway hdfs-site.xml configuration. Then in Flume you can use the remote namespace defined name. See http://community.cloudera.com/t5/Storage-Random-Access-HDFS/distcp-with-same-nameservicename/m-p/49311/highlight/true#M2631 for more details on how to define this.

Re: Flume - HDFS HA

marccasajus — Tue, 23 Feb 2021 12:41:56 GMT

Yes, you can download the hdfs client configuration from Cloudera Manager, but this is not possible always, when you are working on different department or any bureaucratic issue... And if you make any change HDFS configuration, you must download this configuration again. Is not a scalable solution in a big environments, the best solution is working on the same cluster (gateway if is possible), but if you have an external Flume Agents, there no exist a properly and scalable solution I think.