question Re: webhdfs way to always post to ACTIVE namenode in Archives of Support Questions (Read Only)

webhdfs way to always post to ACTIVE namenode

sbn — Fri, 09 Nov 2018 08:59:47 GMT

i have a logging setup where i have a fluent/webhdfs host, for writing logs from fluent to hdfs.

a problem is i can only write to the ACTIVE namenode, this sets some requirements to the logic in the fluent/webhdfs host that i would like to avoid.

is there a way to make sure youre always posting to the ACTIVE namenode?

Re: webhdfs way to always post to ACTIVE namenode

jmanel — Fri, 09 Nov 2018 10:32:13 GMT

It is a fluentd issue, have you checked fluentd documentation ?

Take a look to https://github.com/fluent/fluent-plugin-webhdfs :

<match access.**>
  @type            webhdfs
  namenode         master1.your.cluster.local:50070
  standby_namenode master2.your.cluster.local:50070
  path             /path/on/hdfs/access.log.%Y%m%d_%H.log
</match>