Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Flume - sink values if it is hdfs: http://hortonworks.com/blog/configure-elastic-search-hadoop-hdp-2-0/ "flume example"

avatar
Rising Star

Hi,

I got this link from the before post:

http://hortonworks.com/blog/configure-elastic-search-hadoop-hdp-2-0/ "flume example"

The diagram show the sink can be hdfs or elastic search. I would like to try with hdfs in sandbox.

What should be the value of sink for hdfs. I tried sink.type=logger, the configuration file is throwing an error.

Since I am learning flume by trying out the examples mentioned in sandbox, it would be good if you could help me.

Thank you, Revathy.

1 ACCEPTED SOLUTION

avatar
Master Mentor

Here's all info on Flume sinks and sample hdfs sink configuration

https://flume.apache.org/FlumeUserGuide.html#hdfs-sink

a1.channels = c1
a1.sinks = k1
a1.sinks.k1.type = hdfs
a1.sinks.k1.channel = c1
a1.sinks.k1.hdfs.path = /flume/events/%y-%m-%d/%H%M/%S
a1.sinks.k1.hdfs.filePrefix = events-
a1.sinks.k1.hdfs.round = true
a1.sinks.k1.hdfs.roundValue = 10
a1.sinks.k1.hdfs.roundUnit = minute

View solution in original post

1 REPLY 1

avatar
Master Mentor

Here's all info on Flume sinks and sample hdfs sink configuration

https://flume.apache.org/FlumeUserGuide.html#hdfs-sink

a1.channels = c1
a1.sinks = k1
a1.sinks.k1.type = hdfs
a1.sinks.k1.channel = c1
a1.sinks.k1.hdfs.path = /flume/events/%y-%m-%d/%H%M/%S
a1.sinks.k1.hdfs.filePrefix = events-
a1.sinks.k1.hdfs.round = true
a1.sinks.k1.hdfs.roundValue = 10
a1.sinks.k1.hdfs.roundUnit = minute