Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Flume makes files every 10 minutes

Flume makes files every 10 minutes

New Contributor

Although I have set the Sink to create files every 60 minutes Flume keeps creating files about every 10 minutes.

 

See:

-rw-r--r-- 1 flume nobody  668 Oct 21 13:50 syslog-.1382306401377
-rw-r--r-- 1 flume nobody 1363 Oct 21 14:00 syslog-.1382306401378
-rw-r--r-- 1 flume nobody 1363 Oct 21 14:00 syslog-.1382306401379
-rw-r--r-- 1 flume nobody  668 Oct 21 14:00 syslog-.1382306401380
-rw-r--r-- 1 flume nobody 1185 Oct 21 14:01 syslog-.1382306401381
-rw-r--r-- 1 flume nobody 1138 Oct 21 14:01 syslog-.1382306401382
-rw-r--r-- 1 flume nobody 1185 Oct 21 14:01 syslog-.1382306401383

 

The flume config looks like :

agent.sources = syslogs
agent.channels = syslogChannel
agent.sinks = HdfsSyslogSink

agent.sources.syslogs.type = syslogudp
agent.sources.syslogs.port = 5140
agent.sources.syslogs.host = 161.89.52.152
agent.sources.syslogs.channels = syslogChannel

agent.channels.syslogChannel.type = memory

agent.sinks.HdfsSyslogSink.type = hdfs
agent.sinks.HdfsSyslogSink.channel = syslogChannel
agent.sinks.HdfsSyslogSink.hdfs.path = /flume/events/%Y-%m-%d
agent.sinks.HdfsSyslogSink.hdfs.filePrefix = syslog-
agent.sinks.HdfsSyslogSink.hdfs.round = true
agent.sinks.HdfsSyslogSink.hdfs.roundvalue = 60
agent.sinks.HdfsSyslogSink.hdfs.rountUnit = minute
agent.sinks.HdfsSyslogSink.hdfs.useLocalTimeStamp = true

 

How can I get 1 file per hour?

 

Regards Hans-Peter

1 REPLY 1
Highlighted

Re: Flume makes files every 10 minutes

Cloudera Employee

You need to also set your rollTimeout parameter. That specifies the interval between rolls. Rounding only rounds down the timestamp of the event (in your case it rounds down the event timestamp to the last 10 mins).

Don't have an account?
Coming from Hortonworks? Activate your account here