Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Flume deletePoliy on files marked COMPLETED

Flume deletePoliy on files marked COMPLETED

Explorer

the following agent definition functions perfectly except for one issue.

 

the deletePolicy in "completed_vitals_src1" never deletes the files.

 

Both sources look at the same folder.  "streaming_vitals_src1" ingests new files with a datestamp name (mmddyyyyhh24mi) and renames them to .COMPLETED.  "completed_vitals_src1" has an ignorePattern which ignores files with only digits in the name.  It ingests the data from the files marked ".COMPLETED" 

 

However, once the data is ingested by "completed_vitals_src1" I would expect the files to be deleted.  They are not.

 

Thoughts?

 


streaming_vitals.sources = streaming_vitals_src1 completed_vitals_src1
streaming_vitals.channels = streaming_vitals_channel1 completed_vitals_channel1
streaming_vitals.sinks = streaming_vitals_hdfs_sink1 completed_vitals_hdfs_sink1

# streaming_vitals_src1
streaming_vitals.sources.streaming_vitals_src1.type = spooldir
streaming_vitals.sources.streaming_vitals_src1.channels = streaming_vitals_channel1
streaming_vitals.sources.streaming_vitals_src1.spoolDir = /mnt/windows/ebiz/philips
streaming_vitals.sources.streaming_vitals_src1.inputCharset = ISO-8859-1
streaming_vitals.sources.streaming_vitals_src1.outputCharset = UTF-8
streaming_vitals.sources.streaming_vitals_src1.decodeErrorPolicy = FAIL

# completed_vitals_src1
streaming_vitals.sources.completed_vitals_src1.type = spooldir
streaming_vitals.sources.completed_vitals_src1.channels = completed_vitals_channel1
streaming_vitals.sources.completed_vitals_src1.spoolDir = /mnt/windows/ebiz/philips
streaming_vitals.sources.completed_vitals_src1.ignorePattern = ^\d+$
streaming_vitals.sources.completed_vitals_src1.deletePolicy = immediate
streaming_vitals.sources.completed_vitals_src1.inputCharset = ISO-8859-1
streaming_vitals.sources.completed_vitals_src1.outputCharset = UTF-8
streaming_vitals.sources.completed_vitals_src1.decodeErrorPolicy = FAIL

# streaming_vitals_channel1
streaming_vitals.channels.streaming_vitals_channel1.type = memory
streaming_vitals.channels.streaming_vitals_channel1.capacity = 100000
streaming_vitals.channels.streaming_vitals_channel1.transactionCapacity = 50000

# completed_vitals_channel1
streaming_vitals.channels.completed_vitals_channel1.type = file
streaming_vitals.channels.completed_vitals_channel1.checkpointDir = /store/flume1/streaming_vitals/checkpoint
streaming_vitals.channels.completed_vitals_channel1.dataDirs = /store/flume1/streaming_vitals/data
streaming_vitals.channels.completed_vitals_channel1.capacity = 100000
streaming_vitals.channels.completed_vitals_channel1.transactionCapacity = 50000

# streaming_vitals_hdfs_sink1
streaming_vitals.sinks.streaming_vitals_hdfs_sink1.channel = streaming_vitals_channel1
streaming_vitals.sinks.streaming_vitals_hdfs_sink1.type = hdfs
streaming_vitals.sinks.streaming_vitals_hdfs_sink1.hdfs.writeFormat = Text
streaming_vitals.sinks.streaming_vitals_hdfs_sink1.hdfs.fileType = DataStream
streaming_vitals.sinks.streaming_vitals_hdfs_sink1.hdfs.useLocalTimeStamp = true
streaming_vitals.sinks.streaming_vitals_hdfs_sink1.hdfs.path = /user/choa/philips/streaming_vitals/null_sink
streaming_vitals.sinks.streaming_vitals_hdfs_sink1.hdfs.batchSize = 10000
streaming_vitals.sinks.streaming_vitals_hdfs_sink1.hdfs.rollInterval = 0
streaming_vitals.sinks.streaming_vitals_hdfs_sink1.hdfs.rollCount = 0
streaming_vitals.sinks.streaming_vitals_hdfs_sink1.hdfs.rollSize = 100000
streaming_vitals.sinks.streaming_vitals_hdfs_sink1.hdfs.idleTimeout = 1800

# completed_vitals_hdfs_sink1
streaming_vitals.sinks.completed_vitals_hdfs_sink1.channel = completed_vitals_channel1
streaming_vitals.sinks.completed_vitals_hdfs_sink1.type = hdfs
streaming_vitals.sinks.completed_vitals_hdfs_sink1.hdfs.writeFormat = Text
streaming_vitals.sinks.completed_vitals_hdfs_sink1.hdfs.fileType = DataStream
streaming_vitals.sinks.completed_vitals_hdfs_sink1.hdfs.useLocalTimeStamp = true
streaming_vitals.sinks.completed_vitals_hdfs_sink1.hdfs.path = /user/choa/philips/streaming_vitals/sink
streaming_vitals.sinks.completed_vitals_hdfs_sink1.hdfs.batchSize = 10000
streaming_vitals.sinks.completed_vitals_hdfs_sink1.hdfs.rollInterval = 0
streaming_vitals.sinks.completed_vitals_hdfs_sink1.hdfs.rollCount = 0
streaming_vitals.sinks.completed_vitals_hdfs_sink1.hdfs.rollSize = 100000
streaming_vitals.sinks.completed_vitals_hdfs_sink1.hdfs.idleTimeout = 1800