Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

File Not Found Exception when running Flume to connect with remote FTP Server?

Highlighted

File Not Found Exception when running Flume to connect with remote FTP Server?

Hi All,

@Michael M

I am trying to access the FTP Server log data (text files) from remote system in abroad through FTP client and i followed

https://github.com/keedio/flume-ftp-source

This is my Flume Configuration to access FTP Source logs

### wwww.keedio.com # example file, protocol is ftp, process by lines, and sink to file_roll # for testing purposes. ## Sources Definition for agent "agent" #ACTIVE LIST agent.sources = ftp1 agent.sinks = k1 agent.channels = ch1 ##### SOURCE IS ftp server # Type of source for ftp sources agent.sources.ftp1.type = org.keedio.flume.source.ftp.source.Source agent.sources.ftp1.client.source = ftp # Connection properties for ftp server agent.sources.ftp1.name.server = 192.168.10.23 agent.sources.ftp1.port = 21 agent.sources.ftp1.user = admin agent.sources.ftp1.password = admin@321 agent.sources.ftp1.folder =D:\gprs\ agent.sources.ftp1.file.name = gprs_20161201 # Discover delay, each configured milisecond directory will be explored agent.sources.ftp1.run.discover.delay=5000 # Process by lines agent.sources.ftp1.flushlines = true agent.sinks.k1.type = file_roll agent.sinks.k1.sink.directory = /gprs/live/ agent.sinks.k1.sink.rollInterval = 7200 agent.channels.ch1.type = memory agent.channels.ch1.capacity = 10000 agent.channels.ch1.transactionCapacity = 1000 agent.sources.ftp1.channels = ch1 agent.sinks.k1.channel = ch1

This is Error i am getting

Unable to deliver event. Exception follows. org.apache.flume.EventDeliveryException: Failed to open file /gprs/live/1480650586169-1 while delivering event at org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:183) at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67) at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.FileNotFoundException: /gprs/live/1480650586169-1 (No such file or directory) at java.io.FileOutputStream.open0(Native Method) at java.io.FileOutputStream.open(FileOutputStream.java:270) at java.io.FileOutputStream.<init>(FileOutputStream.java:213) at java.io.FileOutputStream.<init>(FileOutputStream.java:162) at org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:175) ... 3 more 16/12/02 09:19:46 INFO source.Source: Actual dir: / files: 0 16/12/02 09:19:47 INFO source.Source: Discovered: gprs_20160709 ,size: 1402 16/12/02 09:19:48 ERROR client.KeedioSource: Error saving map File java.io.FileNotFoundException: D:agent.sources.ftp1.file.name = gprs_20161202/default_file_track_status.ser (No such file or directory) at java.io.FileOutputStream.open0(Native Method) at java.io.FileOutputStream.open(FileOutputStream.java:270) at java.io.FileOutputStream.<init>(FileOutputStream.java:213) at java.io.FileOutputStream.<init>(FileOutputStream.java:101) at org.keedio.flume.source.ftp.client.KeedioSource.saveMap(KeedioSource.java:220) at org.keedio.flume.source.ftp.source.Source.discoverElements(Source.java:224) at org.keedio.flume.source.ftp.source.Source.process(Source.java:92) at org.apache.flume.source.PollableSourceRunner$PollingRunner.run(PollableSourceRunner.java:133) at java.lang.Thread.run(Thread.java:745) 16/12/02 09:19:48 INFO source.Source: Processed: gprs_20160709 ,total files: 1 16/12/02 09:19:48 INFO source.Source: Discovered: gprs_cdr_20160710 ,size: 485 16/12/02 09:19:49 ERROR client.KeedioSource: Error saving map File java.io.FileNotFoundException: D:agent.sources.ftp1.file.name = gprs_20161202/default_file_track_status.ser (No such file or directory) at java.io.FileOutputStream.open0(Native Method) at java.io.FileOutputStream.open(FileOutputStream.java:270) at java.io.FileOutputStream.<init>(FileOutputStream.java:213) at java.io.FileOutputStream.<init>(FileOutputStream.java:101) at org.keedio.flume.source.ftp.client.KeedioSource.saveMap(KeedioSource.java:220) at org.keedio.flume.source.ftp.source.Source.discoverElements(Source.java:224) at org.keedio.flume.source.ftp.source.Source.process(Source.java:92) at org.apache.flume.source.PollableSourceRunner$PollingRunner.run(PollableSourceRunner.java:133) at java.lang.Thread.run(Thread.java:745) 16/12/02 09:19:49 INFO source.Source: Processed: gprs_20160710 ,total files: 2 16/12/02 09:19:49 INFO source.Source: Discovered: gprs_20160713 ,size: 1857 16/12/02 09:19:50 ERROR client.KeedioSource: Error saving map File java.io.FileNotFoundException: D:agent.sources.ftp1.file.name = gprs_20161202/default_file_track_status.ser (No such file or directory) at java.io.FileOutputStream.open0(Native Method) at java.io.FileOutputStream.open(FileOutputStream.java:270) at java.io.FileOutputStream.<init>(FileOutputStream.java:213) at java.io.FileOutputStream.<init>(FileOutputStream.java:101) at org.keedio.flume.source.ftp.client.KeedioSource.saveMap(KeedioSource.java:220) at org.keedio.flume.source.ftp.source.Source.discoverElements(Source.java:224) at org.keedio.flume.source.ftp.source.Source.process(Source.java:92) at org.apache.flume.source.PollableSourceRunner$PollingRunner.run(PollableSourceRunner.java:133) at java.lang.Thread.run(Thread.java:745) 16/12/02 09:19:50 INFO source.Source: Processed: gprs_20160713 ,total files: 3 16/12/02 09:19:51 INFO source.Source: Discovered: gprs_cdr_20160714 ,size: 56427 16/12/02 09:19:51 ERROR flume.SinkRunner: Unable to deliver event. Exception follows. org.apache.flume.EventDeliveryException: Failed to open file /gprs/live/1480650586169-1 while delivering event at org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:183) at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67) at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.FileNotFoundException: /gprs/live/1480650586169-1 (No such file or directory) at java.io.FileOutputStream.open0(Native Method) at java.io.FileOutputStream.open(FileOutputStream.java:270) at java.io.FileOutputStream.<init>(FileOutputStream.java:213) at java.io.FileOutputStream.<init>(FileOutputStream.java:162) at org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:175) ... 3 more 16/12/02 09:19:52 ERROR client.KeedioSource: Error saving map File java.io.FileNotFoundException: D:agent.sources.ftp1.file.name = gprs_cdr_20161202/default_file_track_status.ser (No such file or directory) at java.io.FileOutputStream.open0(Native Method) at java.io.FileOutputStream.open(FileOutputStream.java:270) at java.io.FileOutputStream.<init>(FileOutputStream.java:213) at java.io.FileOutputStream.<init>(FileOutputStream.java:101) at org.keedio.flume.source.ftp.client.KeedioSource.saveMap(KeedioSource.java:220) at org.keedio.flume.source.ftp.source.Source.discoverElements(Source.java:224) at org.keedio.flume.source.ftp.source.Source.process(Source.java:92) at org.apache.flume.source.PollableSourceRunner$PollingRunner.run(PollableSourceRunner.java:133) at java.lang.Thread.run(Thread.java:745) 16/12/02 09:19:52 INFO source.Source: Processed: gprs_cdr_20160714 ,total files: 4 16/12/02 09:19:52 INFO source.Source: Discovered: gprs_cdr_20160715 ,size: 75901 16/12/02 09:19:54 ERROR client.KeedioSource: Error saving map File java.io.FileNotFoundException: D:agent.sources.ftp1.file.name = gprs_20161202/default_file_track_status.ser (No such file or directory) at java.io.FileOutputStream.open0(Native Method) at java.io.FileOutputStream.open(FileOutputStream.java:270) at java.io.FileOutputStream.<init>(FileOutputStream.java:213) at java.io.FileOutputStream.<init>(FileOutputStream.java:101) at org.keedio.flume.source.ftp.client.KeedioSource.saveMap(KeedioSource.java:220) at org.keedio.flume.source.ftp.source.Source.discoverElements(Source.java:224) at org.keedio.flume.source.ftp.source.Source.process(Source.java:92) at org.apache.flume.source.PollableSourceRunner$PollingRunner.run(PollableSourceRunner.java:133) at java.lang.Thread.run(Thread.java:745) 16/12/02 09:19:54 INFO source.Source: Processed: gprs_20160715 ,total files: 5 16/12/02 09:19:54 INFO source.Source: Discovered: gprs_20160716 ,size: 63249 16/12/02 09:19:55 ERROR client.KeedioSource: Error saving map File java.io.FileNotFoundException: D:agent.sources.ftp1.file.name = gprs_20161202/default_file_track_status.ser (No such file or directory) at java.io.FileOutputStream.open0(Native Method) at java.io.FileOutputStream.open(FileOutputStream.java:270) at java.io.FileOutputStream.<init>(FileOutputStream.java:213) at java.io.FileOutputStream.<init>(FileOutputStream.java:101) at org.keedio.flume.source.ftp.client.KeedioSource.saveMap(KeedioSource.java:220) at org.keedio.flume.source.ftp.source.Source.discoverElements(Source.java:224) at org.keedio.flume.source.ftp.source.Source.process(Source.java:92) at org.apache.flume.source.PollableSourceRunner$PollingRunner.run(PollableSourceRunner.java:133) at java.lang.Thread.run(Thread.java:745) 16/12/02 09:19:55 INFO source.Source: Processed: gprs_20160716 ,total files: 6 16/12/02 09:19:56 INFO source.Source: Discovered: gprs_20160717 ,size: 465 16/12/02 09:19:56 ERROR flume.SinkRunner: Unable to deliver event. Exception follows. org.apache.flume.EventDeliveryException: Failed to open file /gprs/live/1480650586169-1 while delivering event at org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:183) at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67) at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.FileNotFoundException: /gprs/live/1480650586169-1 (No such file or directory) at java.io.FileOutputStream.open0(Native Method) at java.io.FileOutputStream.open(FileOutputStream.java:270) at java.io.FileOutputStream.<init>(FileOutputStream.java:213) at java.io.FileOutputStream.<init>(FileOutputStream.java:162) at org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:175) ... 3 more 16/12/02 09:19:56 ERROR client.KeedioSource: Error saving map File java.io.FileNotFoundException: D:agent.sources.ftp1.file.name = gprs_20161202/default_file_track_status.ser (No such file or directory) at java.io.FileOutputStream.open0(Native Method) at java.io.FileOutputStream.open(FileOutputStream.java:270) at java.io.FileOutputStream.<init>(FileOutputStream.java:213) at java.io.FileOutputStream.<init>(FileOutputStream.java:101) at org.keedio.flume.source.ftp.client.KeedioSource.saveMap(KeedioSource.java:220) at org.keedio.flume.source.ftp.source.Source.discoverElements(Source.java:224) at org.keedio.flume.source.ftp.source.Source.process(Source.java:92) at org.apache.flume.source.PollableSourceRunner$PollingRunner.run(PollableSourceRunner.java:133) at java.lang.Thread.run(Thread.java:745) 16/12/02 09:19:56 INFO source.Source: Processed: gprs_20160717 ,total files: 7 16/12/02 09:19:57 INFO source.Source: Discovered: gprs_cdr_20160718 ,size: 53289 16/12/02 09:19:58 ERROR client.KeedioSource: Error saving map File java.io.FileNotFoundException: D:agent.sources.ftp1.file.name = gprs_20161202/default_file_track_status.ser (No such file or directory) at java.io.FileOutputStream.open0(Native Method) at java.io.FileOutputStream.open(FileOutputStream.java:270) at java.io.FileOutputStream.<init>(FileOutputStream.java:213) at java.io.FileOutputStream.<init>(FileOutputStream.java:101) at org.keedio.flume.source.ftp.client.KeedioSource.saveMap(KeedioSource.java:220) at org.keedio.flume.source.ftp.source.Source.discoverElements(Source.java:224) at org.keedio.flume.source.ftp.source.Source.process(Source.java:92) at org.apache.flume.source.PollableSourceRunner$PollingRunner.run(PollableSourceRunner.java:133) at java.lang.Thread.run(Thread.java:745) 16/12/02 09:19:58 INFO source.Source: Processed: gprs_cdr_20160718 ,total files: 8 16/12/02 09:19:58 INFO source.Source: Discovered: gprs_cdr_20160719 ,size: 128436 16/12/02 09:20:00 ERROR client.KeedioSource: Error saving map File java.io.FileNotFoundException: D:agent.sources.ftp1.file.name = gprs_20161202/default_file_track_status.ser (No such file or directory) at java.io.FileOutputStream.open0(Native Method) at java.io.FileOutputStream.open(FileOutputStream.java:270) at java.io.FileOutputStream.<init>(FileOutputStream.java:213) at java.io.FileOutputStream.<init>(FileOutputStream.java:101) at org.keedio.flume.source.ftp.client.KeedioSource.saveMap(KeedioSource.java:220) at org.keedio.flume.source.ftp.source.Source.discoverElements(Source.java:224) at org.keedio.flume.source.ftp.source.Source.process(Source.java:92) at org.apache.flume.source.PollableSourceRunner$PollingRunner.run(PollableSourceRunner.java:133) at java.lang.Thread.run(Thread.java:745) 16/12/02 09:20:00 INFO source.Source: Processed: gprs_20160719 ,total files: 9

This is error i am facing for the past few days and trying to resolve it.

Thanks in Advance

6 REPLIES 6

Re: File Not Found Exception when running Flume to connect with remote FTP Server?

Explorer

Agent configuration is loaded by java.util.Properties where the backslash is the escape caracter. In your configuration the following two lines:

agent.sources.ftp1.folder =D:\gprs\
agent.sources.ftp1.file.name = gprs_20161201

Loaded as one property:

Name = "agent.sources.ftp1.folder"
Value = "D:gprsagent.sources.ftp1.file.name = gprs_20161201"

This happens because at the line end the backslash character escapes the newline causing that everything until the next newline character will belong to the first property value. To fix this you have to escape the backslashes like this:

agent.sources.ftp1.folder = D:\\gprs\\
agent.sources.ftp1.file.name = gprs_20161201

Re: File Not Found Exception when running Flume to connect with remote FTP Server?

@mhegedus

Hi mhegedus,

When i changed to two backslashes and i am getting the below error when trying to access the D Folder in FTP Server which acts the source. Whether we need to change the configuration.

Exception 1

16/12/08 19:26:44 ERROR source.Source: Folder D:\gprs\ not exists

Exception 2

org.apache.flume.EventDeliveryException: Failed to open file /gprs/live/1481205405309-1 while delivering event at org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:183) at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67) at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.FileNotFoundException: /gprs/live/1481205405309-1 (No such file or directory) at java.io.FileOutputStream.open0(Native Method) at java.io.FileOutputStream.open(FileOutputStream.java:270) at java.io.FileOutputStream.<init>(FileOutputStream.java:213) at java.io.FileOutputStream.<init>(FileOutputStream.java:162) at org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:175)

Before fetching the data in HDFS, it is trying to access the

/gprs/live/1481205405309-1 (No such file or directory) (HDFS PATH)

But such file is not created in HDFS

I followed this link

https://github.com/keedio/flume-ftp-source

and build the jar. I tried in Google but its pointing to the same Keedio FTP Source link.

If you have worked on Flume FTP Source, please provide the steps to fetch the real time data to hadoop using flume. If i need include any other jar to achieve the result.

please help me to resolve the issue to fetch the real time data into hadoop.

Re: File Not Found Exception when running Flume to connect with remote FTP Server?

Expert Contributor

Hi,

agent.sinks.k1.sink.directory = /gprs/live/

This directory should exist.

Re: File Not Found Exception when running Flume to connect with remote FTP Server?

Explorer

Exception 1: The path might be different from "D:\gprs\" when you access your ftp server, for example: "/gprs" depending on what is the root directory.

Exception 2: File roll sink (agent.sinks.k1.type = file_roll) will not save data to hdfs but to the local filesystem where the agent runs.

Re: File Not Found Exception when running Flume to connect with remote FTP Server?

@Avijeet Dash

Hi Avijeet,

I have already created the /gprs/live in hdfs but also it shows the file not found exception. please do the needful

Re: File Not Found Exception when running Flume to connect with remote FTP Server?

Explorer

@Magesh Kumar, please edit the question and format the stack trace as code with line feeds and truncate it to contain the exception only once. Could also mark this question as resolved?

Don't have an account?
Coming from Hortonworks? Activate your account here