question Re: How to put files in flume spooldir one by one in Archives of Support Questions (Read Only)

How to put files in flume spooldir one by one

indranil — Fri, 16 Sep 2022 10:35:49 GMT

i am using flume spooldir to put files in HDFS , but i am getting so many small files in HDFS. I thought of using batch size and roll interval but i don't want to get dependent on size and interval. So I decided to push files in flume spooldir one at a time. How can i do this ? Please help

Re: How to put files in flume spooldir one by one

csguna — Thu, 01 Sep 2016 06:54:35 GMT

Try using the timestamp interceptor.

Re: How to put files in flume spooldir one by one

indranil — Thu, 01 Sep 2016 06:57:19 GMT

any examples you have ?

Re: How to put files in flume spooldir one by one

csguna — Thu, 01 Sep 2016 07:05:15 GMT

You can refer Hdfs sink timestamp escape sequence , there is alot of them you can use accordingly .

example

U can use hdfs bucketing , for every one hour.

agen1.sinks.hdfsSinks.hdfs.path = /data/flume/%{aa}/%y/%m/%d/%H/%M
agent1.sinks.hdfsSinks.hdfs.round = true
agen1.sinks.hdfsSinks.roundUnit = hour
agen1.sinks.hdfsSinks.roundValue = 1