question Re: Flume error while testing spooldir source in Archives of Support Questions (Read Only)

Flume error while testing spooldir source

rpulluru — Fri, 30 Jun 2017 20:34:02 GMT

While testing Flume spooldir source, I am getting this error in flume logs.

28 Jun 2017 15:06:44,560 ERROR [pool-7-thread-1] (org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run:280) - FATAL: 
Spool Directory source spooldir-source: { spoolDir: /data/src/input }: Uncaught exception in SpoolDirectorySource thread. 
Restart or reconfigure Flume to continue processing. 
java.lang.IllegalStateException: File has changed size since being read: /home/userapp/test/newfile_5_1.csv.gz 
at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.retireCurrentFile(ReliableSpoolingFileEventReader.java:410) 
at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:326) 
at org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:250) 
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) 
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) 
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) 
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) 
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
at java.lang.Thread.run(Thread.java:745)

Any pointers?

Re: Flume error while testing spooldir source

Schandhok — Fri, 30 Jun 2017 20:37:56 GMT

@rpulluru

This issue occurs when one of the following is true:

If a file is written to after being placed into the spooling directory, Flume will print an error to its log file and stop processing.
If a file name is reused at a later time, Flume will print an error to its log file and stop processing.

If you are copying the files in your /data/src/input directory, change the operation to ‘mv’, Or you can copy the files as .tmp and then 'mv' the '.tmp' file to the same spooling directory with the actual name. Add the following line in flume.conf to ignore .tmp files in SpoolDir:

Agent1.sources.spooldir-source.ignorePattern=^.*\.tmp$

Re: Flume error while testing spooldir source

rpulluru — Fri, 30 Jun 2017 21:25:04 GMT

@Shashank Chandhok Thanks. I was copying the files. Moving them helped.