Spooldir Error in Flume

Vinod369 — Fri, 16 Sep 2022 10:25:35 GMT

I am getting below error while running Flume agent to spool directory,

2016-06-16 10:07:35,927 INFO org.apache.flume.source.SpoolDirectorySource: SpoolDirectorySource source starting with directory: /export/home/user1/flume/input
2016-06-16 10:07:35,927 ERROR org.apache.flume.lifecycle.LifecycleSupervisor: Unable to start EventDrivenSourceRunner: { source:Spool Directory source sdir: { spoolDir: /export/home/user1/flume/input } } - Exception follows.
java.lang.IllegalStateException: Directory does not exist: /export/home/user1/flume/input
        at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
        at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.<init>(ReliableSpoolingFileEventReader.java:145)
        at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.<init>(ReliableSpoolingFileEventReader.java:77)
        at org.apache.flume.client.avro.ReliableSpoolingFileEventReader$Builder.build(ReliableSpoolingFileEventReader.java:669)
        at org.apache.flume.source.SpoolDirectorySource.start(SpoolDirectorySource.java:85)
        at org.apache.flume.source.EventDrivenSourceRunner.start(EventDrivenSourceRunner.java:44)
        at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

- I am running Flume as service on Edge node.

- Directory '/export/home/user1/flume/input' is on same Edge node, but this direcory is owned by user1

- I assume Flume agent on Edge node run by 'Flume' as user

- May be error is because 'Flume' user not having r / w access on folder '/export/home/user1/flume/input'

If my understanding is correct then I must create a spool directory where Flume as user have r/w access and other users posting files in that folder have at-least write permission.

Please correct me if my understanding is not correct.

Re: Spooldir Error in Flume

pdvorak — Fri, 24 Jun 2016 15:34:30 GMT

Your understanding is correct. You either need to ensure flume can write to that directory, or create a directory that flume owns and can write to.

-pd

question Re: Spooldir Error in Flume in Archives of Support Questions (Read Only)

Spooldir Error in Flume

Re: Spooldir Error in Flume