Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Can we use variable hostname for flume sink configuration inside flume config file?

avatar
Rising Star

Given a hfds sink configuration :

ale.sinks.sink1.hdfs.path = hdfs://ham-dal-0001.corp.wayport.net:8020/prod/hadoop/smallsite/flume_ingest_ale2_hak_3/station/%Y/%m/%d/%H

The question is that can we use {variable hostname} in flume.conf file. For example:

ale.sinks.sink1.hdfs.path = hdfs://ham-dal-0001.corp.wayport.net:8020/prod/hadoop/smallsite/{variable hostname}/station/%Y/%m/%d/%H 						

Thank you.

1 ACCEPTED SOLUTION

avatar

Flume provides a Host interceptor that will inserts a header with key host or a configured key whose value is the hostname or IP address of the host, based on configuration. If you want to set a specific key value in the header you can use the Static Interceptor. You should be able to use %{host} in a latter sink. If the host name that you want to add is the hostname where the agent is running and its an HDFS sink you can use %{host} directly without the need of interceptor, see here.

View solution in original post

2 REPLIES 2

avatar

Flume provides a Host interceptor that will inserts a header with key host or a configured key whose value is the hostname or IP address of the host, based on configuration. If you want to set a specific key value in the header you can use the Static Interceptor. You should be able to use %{host} in a latter sink. If the host name that you want to add is the hostname where the agent is running and its an HDFS sink you can use %{host} directly without the need of interceptor, see here.

avatar
Rising Star

@deepesh@hortonworks.com

This is a good information. Thank you.