Support Questions
Find answers, ask questions, and share your expertise

Can we use variable hostname for flume sink configuration inside flume config file?

Solved Go to solution

Can we use variable hostname for flume sink configuration inside flume config file?

Contributor

Given a hfds sink configuration :

ale.sinks.sink1.hdfs.path = hdfs://ham-dal-0001.corp.wayport.net:8020/prod/hadoop/smallsite/flume_ingest_ale2_hak_3/station/%Y/%m/%d/%H

The question is that can we use {variable hostname} in flume.conf file. For example:

ale.sinks.sink1.hdfs.path = hdfs://ham-dal-0001.corp.wayport.net:8020/prod/hadoop/smallsite/{variable hostname}/station/%Y/%m/%d/%H 						

Thank you.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Can we use variable hostname for flume sink configuration inside flume config file?

Master Collaborator

Flume provides a Host interceptor that will inserts a header with key host or a configured key whose value is the hostname or IP address of the host, based on configuration. If you want to set a specific key value in the header you can use the Static Interceptor. You should be able to use %{host} in a latter sink. If the host name that you want to add is the hostname where the agent is running and its an HDFS sink you can use %{host} directly without the need of interceptor, see here.

View solution in original post

2 REPLIES 2

Re: Can we use variable hostname for flume sink configuration inside flume config file?

Master Collaborator

Flume provides a Host interceptor that will inserts a header with key host or a configured key whose value is the hostname or IP address of the host, based on configuration. If you want to set a specific key value in the header you can use the Static Interceptor. You should be able to use %{host} in a latter sink. If the host name that you want to add is the hostname where the agent is running and its an HDFS sink you can use %{host} directly without the need of interceptor, see here.

View solution in original post

Re: Can we use variable hostname for flume sink configuration inside flume config file?

Contributor

@deepesh@hortonworks.com

This is a good information. Thank you.