Can we use variable hostname for flume sink configuration inside flume config file?

[#1]

Given a hfds sink configuration :

ale.sinks.sink1.hdfs.path = hdfs://ham-dal-0001.corp.wayport.net:8020/prod/hadoop/smallsite/flume_ingest_ale2_hak_3/station/%Y/%m/%d/%H

The question is that can we use {variable hostname} in flume.conf file. For example:

ale.sinks.sink1.hdfs.path = hdfs://ham-dal-0001.corp.wayport.net:8020/prod/hadoop/smallsite/{variable hostname}/station/%Y/%m/%d/%H 						

Thank you.

[#2 · Accepted solution]

Flume provides a Host interceptor that will inserts a header with key host or a configured key whose value is the hostname or IP address of the host, based on configuration. If you want to set a specific key value in the header you can use the Static Interceptor. You should be able to use %{host} in a latter sink. If the host name that you want to add is the hostname where the agent is running and its an HDFS sink you can use %{host} directly without the need of interceptor, see here.

[#3]

@[email protected]

This is a good information. Thank you.