11-09-2018 12:59 AM
i have a logging setup where i have a fluent/webhdfs host, for writing logs from fluent to hdfs.
a problem is i can only write to the ACTIVE namenode, this sets some requirements to the logic in the fluent/webhdfs host that i would like to avoid.
is there a way to make sure youre always posting to the ACTIVE namenode?
11-09-2018 02:32 AM
It is a fluentd issue, have you checked fluentd documentation ?
Take a look to https://github.com/fluent/fluent-plugin-webhdfs :
<match access.**> @type webhdfs namenode master1.your.cluster.local:50070 standby_namenode master2.your.cluster.local:50070 path /path/on/hdfs/access.log.%Y%m%d_%H.log </match>