question Can we use variable hostname for flume sink configuration inside flume config file? in Archives of Support Questions (Read Only)

Can we use variable hostname for flume sink configuration inside flume config file?

vjiang — Mon, 05 Oct 2015 05:03:49 GMT

Given a hfds sink configuration :

ale.sinks.sink1.hdfs.path = hdfs://ham-dal-0001.corp.wayport.net:8020/prod/hadoop/smallsite/flume_ingest_ale2_hak_3/station/%Y/%m/%d/%H

The question is that can we use {variable hostname} in flume.conf file. For example:

ale.sinks.sink1.hdfs.path = hdfs://ham-dal-0001.corp.wayport.net:8020/prod/hadoop/smallsite/{variable hostname}/station/%Y/%m/%d/%H

Thank you.

Re: Can we use variable hostname for flume sink configuration inside flume config file?

deepesh1 — Mon, 05 Oct 2015 05:22:23 GMT

Flume provides a Host interceptor that will inserts a header with key host or a configured key whose value is the hostname or IP address of the host, based on configuration. If you want to set a specific key value in the header you can use the Static Interceptor. You should be able to use %{host} in a latter sink. If the host name that you want to add is the hostname where the agent is running and its an HDFS sink you can use %{host} directly without the need of interceptor, see here.

Re: Can we use variable hostname for flume sink configuration inside flume config file?

vjiang — Tue, 06 Oct 2015 05:28:23 GMT

@deepesh@hortonworks.com

This is a good information. Thank you.