Reply
New Contributor
Posts: 3
Registered: ‎10-21-2013

Flume makes files every 10 minutes

Although I have set the Sink to create files every 60 minutes Flume keeps creating files about every 10 minutes.

 

See:

-rw-r--r-- 1 flume nobody  668 Oct 21 13:50 syslog-.1382306401377
-rw-r--r-- 1 flume nobody 1363 Oct 21 14:00 syslog-.1382306401378
-rw-r--r-- 1 flume nobody 1363 Oct 21 14:00 syslog-.1382306401379
-rw-r--r-- 1 flume nobody  668 Oct 21 14:00 syslog-.1382306401380
-rw-r--r-- 1 flume nobody 1185 Oct 21 14:01 syslog-.1382306401381
-rw-r--r-- 1 flume nobody 1138 Oct 21 14:01 syslog-.1382306401382
-rw-r--r-- 1 flume nobody 1185 Oct 21 14:01 syslog-.1382306401383

 

The flume config looks like :

agent.sources = syslogs
agent.channels = syslogChannel
agent.sinks = HdfsSyslogSink

agent.sources.syslogs.type = syslogudp
agent.sources.syslogs.port = 5140
agent.sources.syslogs.host = 161.89.52.152
agent.sources.syslogs.channels = syslogChannel

agent.channels.syslogChannel.type = memory

agent.sinks.HdfsSyslogSink.type = hdfs
agent.sinks.HdfsSyslogSink.channel = syslogChannel
agent.sinks.HdfsSyslogSink.hdfs.path = /flume/events/%Y-%m-%d
agent.sinks.HdfsSyslogSink.hdfs.filePrefix = syslog-
agent.sinks.HdfsSyslogSink.hdfs.round = true
agent.sinks.HdfsSyslogSink.hdfs.roundvalue = 60
agent.sinks.HdfsSyslogSink.hdfs.rountUnit = minute
agent.sinks.HdfsSyslogSink.hdfs.useLocalTimeStamp = true

 

How can I get 1 file per hour?

 

Regards Hans-Peter

Highlighted
Cloudera Employee
Posts: 8
Registered: ‎07-30-2013

Re: Flume makes files every 10 minutes

You need to also set your rollTimeout parameter. That specifies the interval between rolls. Rounding only rounds down the timestamp of the event (in your case it rounds down the event timestamp to the last 10 mins).

Announcements
New solutions