I am getting below error while running Flume agent to spool directory, 2016-06-16 10:07:35,927 INFO org.apache.flume.source.SpoolDirectorySource: SpoolDirectorySource source starting with directory: /export/home/user1/flume/input 2016-06-16 10:07:35,927 ERROR org.apache.flume.lifecycle.LifecycleSupervisor: Unable to start EventDrivenSourceRunner: { source:Spool Directory source sdir: { spoolDir: /export/home/user1/flume/input } } - Exception follows. java.lang.IllegalStateException: Directory does not exist: /export/home/user1/flume/input         at com.google.common.base.Preconditions.checkState(Preconditions.java:145)         at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.<init>(ReliableSpoolingFileEventReader.java:145)         at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.<init>(ReliableSpoolingFileEventReader.java:77)         at org.apache.flume.client.avro.ReliableSpoolingFileEventReader$Builder.build(ReliableSpoolingFileEventReader.java:669)         at org.apache.flume.source.SpoolDirectorySource.start(SpoolDirectorySource.java:85)         at org.apache.flume.source.EventDrivenSourceRunner.start(EventDrivenSourceRunner.java:44)         at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)         at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   - I am running Flume as service on Edge node. - Directory '/export/home/user1/flume/input' is on same Edge node, but this direcory is owned by user1 - I assume Flume agent on Edge node run by 'Flume' as user  - May be error is because 'Flume' user not having r / w access on folder  '/export/home/user1/flume/input'    If my understanding is correct then I must create a spool directory where Flume as user have r/w access and other users posting files in that folder have at-least write permission.   Please correct me if my understanding is not correct.   ... View more

I am trying to read the data from SQL Server using REST API as a Flume source with JSON format. Not sure which Flume source I can use here (may be HTTP) ?   If it is HTTP then where I should mention the API URL(http://api.xxxx.com.xxxx) in Flume source property ? Please advice. ... View more

Hi, I have below questions related to Flume, On which node should Flume agent run ? On Edge node or one of Hadoop cluster node ? Do I need to run Flume agent using nohup in production as it may keep running until interrupted ... View more

Hi, Encountered below error while connecting to hive2 using SAS, ERROR: Unable to connect to the Hive server. ERROR: Error trying to establish connection. ERROR: Error in the LIBNAME statement.    Same user can connect to hive but to different schema.   ... View more