List/fetch ftp/sftp processors problem

[#1]

ftp/sftp processors don't work....

also i can't stop them without restrarting NIFI app.

thats what i see in app.log

2017-06-12 13:37:42,892 INFO [StandardProcessScheduler Thread-7] o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled ListFTP[id=0b26dd50-015c-1000-ab83-c3df49136530] to run with 1 threads 2017-06-12 13:47:20,337 INFO [NiFi Web Server-18214] o.a.n.controller.StandardProcessorNode Stopping processor: class org.apache.nifi.processors.standard.ListFTP 2017-06-12 13:47:20,337 INFO [StandardProcessScheduler Thread-6] o.a.n.c.s.TimerDrivenSchedulingAgent Stopped scheduling ListFTP[id=0b26dd50-015c-1000-ab83-c3df49136530] to run 2017-06-12 13:50:46,761 WARN [StandardProcessScheduler Thread-3] o.a.n.controller.StandardProcessorNode Failed while shutting down processor ListFTP[id=0b26dd50-015c-1000-ab83-c3df49136530] 2017-06-12 13:50:46,762 WARN [StandardProcessScheduler Thread-4] o.a.n.controller.StandardProcessorNode Failed while shutting down processor ListSFTP[id=196c3fff-9d59-1fed-9902-5daf3757977f] 2017-06-12 13:50:46,762 WARN [StandardProcessScheduler Thread-8] o.a.n.controller.StandardProcessorNode Failed while shutting down processor ListFTP[id=0b26dd50-015c-1000-0000-000003bb95df] 2017-06-12 13:50:46,762 WARN [StandardProcessScheduler Thread-2] o.a.n.controller.StandardProcessorNode Failed while shutting down processor ListSFTP[id=196c3fff-9d59-1fed-8d2d-adbbcbd1c699] 2017-06-12 13:50:46,803 ERROR [Timer-Driven Process Thread-17] o.a.nifi.processors.standard.ListFTP ListFTP[id=0b26dd50-015c-1000-ab83-c3df49136530] ListFTP[id=0b26dd50-015c-1000-ab83-c3df49136530] failed to process due to java.lang.IllegalStateException: Partition is closed; rolling back session: java.lang.IllegalStateException: Partition is closed 2017-06-12 13:50:46,812 ERROR [Timer-Driven Process Thread-17] o.a.nifi.processors.standard.ListFTP 2017-06-12 13:50:46,813 ERROR [Timer-Driven Process Thread-17] o.a.nifi.processors.standard.ListFTP ListFTP[id=0b26dd50-015c-1000-ab83-c3df49136530] ListFTP[id=0b26dd50-015c-1000-ab83-c3df49136530] failed to process session due to java.lang.IllegalStateException: Partition is closed: java.lang.IllegalStateException: Partition is closed 2017-06-12 13:50:46,814 ERROR [Timer-Driven Process Thread-17] o.a.nifi.processors.standard.ListFTP 2017-06-12 13:50:46,814 WARN [Timer-Driven Process Thread-17] o.a.nifi.processors.standard.ListFTP ListFTP[id=0b26dd50-015c-1000-ab83-c3df49136530] Processor Administratively Yielded for 1 sec due to processing failure 2017-06-12 13:50:46,814 WARN [Timer-Driven Process Thread-17] o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding ListFTP[id=0b26dd50-015c-1000-ab83-c3df49136530] due to uncaught Exception: java.lang.IllegalStateException: Partition is closed org.apache.nifi.processors.standard.GetSFTP || org.apache.nifi.nar.NarClassLoader[/nifi/work/nar/extensions/nifi-standard-nar-1.1.0.2.1.2.0-10.nar-unpacked] org.apache.nifi.processors.standard.FetchFTP || org.apache.nifi.nar.NarClassLoader[/nifi/work/nar/extensions/nifi-standard-nar-1.1.0.2.1.2.0-10.nar-unpacked] org.apache.nifi.processors.standard.PutFTP || org.apache.nifi.nar.NarClassLoader[/nifi/work/nar/extensions/nifi-standard-nar-1.1.0.2.1.2.0-10.nar-unpacked] org.apache.nifi.processors.standard.PutSFTP || org.apache.nifi.nar.NarClassLoader[/nifi/work/nar/extensions/nifi-standard-nar-1.1.0.2.1.2.0-10.nar-unpacked] org.apache.nifi.processors.standard.ListFTP || org.apache.nifi.nar.NarClassLoader[/nifi/work/nar/extensions/nifi-standard-nar-1.1.0.2.1.2.0-10.nar-unpacked] org.apache.nifi.processors.standard.FetchSFTP || org.apache.nifi.nar.NarClassLoader[/nifi/work/nar/extensions/nifi-standard-nar-1.1.0.2.1.2.0-10.nar-unpacked] org.apache.nifi.processors.standard.GetFTP || org.apache.nifi.nar.NarClassLoader[/nifi/work/nar/extensions/nifi-standard-nar-1.1.0.2.1.2.0-10.nar-unpacked] org.apache.nifi.processors.standard.ListSFTP || org.apache.nifi.nar.NarClassLoader[/nifi/work/nar/extensions/nifi-standard-nar-1.1.0.2.1.2.0-10.nar-unpacked] 2017-06-12 13:53:46,940 INFO [StandardProcessScheduler Thread-8] o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled ListFTP[id=0b26dd50-015c-1000-ab83-c3df49136530] to run with 1 threads 2017-06-12 13:53:51,155 INFO [NiFi Web Server-50] o.a.n.controller.StandardProcessorNode Stopping processor: class org.apache.nifi.processors.standard.ListFTP 2017-06-12 13:53:51,158 INFO [StandardProcessScheduler Thread-2] o.a.n.c.s.TimerDrivenSchedulingAgent Stopped scheduling ListFTP[id=0b26dd50-015c-1000-ab83-c3df49136530] to run

[#2]

2017-06-12 16:53:25,034 WARN [main] org.apache.hadoop.util.NativeCodeLoader Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2017-06-12 16:53:33,172 WARN [StandardProcessScheduler Thread-1] org.apache.hadoop.util.NativeCodeLoader Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

[#3]

Hello @Oleksandr Solomko

Sorry to hear that you had an issue with FTP processors.

Would you elaborate the symptoms so that we can diagnose better? When you said "ftp/sftp processors don't work", do you see a number (usually 1 or 2) at the right top corner of the processor keeps being shown? If so, the processor may be stuck at some point while NiFi scheduler executes it. In this case, NiFi thread dump will be informative to nail down what the processor is doing or where it gets stuck. To take thread dump, please execute following command:

${NIFI_HOME}/bin/nifi.sh dump

Then Java stack trace will be logged into logs/nifi-bootstrap.log.

It seems the logs you shared previously are ones that were logged when you stopped ListFTP processor (excuse me if I'm wrong). I found this log particularly interesting, this message looks like ListFTP was in the middle of doing something when it's told to stop.

2017-06-12 13:50:46,803 ERROR [Timer-Driven Process Thread-17] o.a.nifi.processors.standard.LisFTP ListFTP[id=0b26dd50-015c-1000-ab83-c3df49136530] ListFTP[id=0b26dd50-015c-1000-ab83-c3df49136530] failed to process due to java.lang.IllegalStateException: Partition is closed; rolling back session: java.lang.IllegalStateException: Partition is closed

Thread dump would be helpful to know what is happening. Please run nifi.sh dump while ListFTP is running (and seems not working, before shutting NiFi process down), and share nifi-bootstrap.log with us if possible.

Thanks,

Koji

[#4]

Hi, bootstrap.txt and screen are in attach

bootstrap.txt

Also I have new errors

2017-06-13 11:24:20,604 ERROR [Timer-Driven Process Thread-34] o.a.nifi.processors.standard.FetchFTP FetchFTP[id=0b272963-015c-1000-b879-09a07fe75263] Failed to fetch content for StandardFlowFileRecord[uuid=f24fe72d-496b-4f07-9f6e-88368c9b59de,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1497338630546-1, container=default, section=1], offset=0, length=1022734],offset=0,name=TIAGPRSVMPT063246,size=1022734] from filename /edb4tel/data/DMP/TIAGPRSVMPT063246 on remote host 172.30.129.1:21 due to java.net.SocketTimeoutException: Read timed out; routing to failure: java.net.SocketTimeoutException: Read timed out 2017-06-13 11:24:20,605 ERROR [Timer-Driven Process Thread-34] o.a.nifi.processors.standard.FetchFTP java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) ~[na:1.8.0_77] at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) ~[na:1.8.0_77] at java.net.SocketInputStream.read(SocketInputStream.java:170) ~[na:1.8.0_77] at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[na:1.8.0_77] at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284) ~[na:1.8.0_77] at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326) ~[na:1.8.0_77] at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178) ~[na:1.8.0_77] at java.io.InputStreamReader.read(InputStreamReader.java:184) ~[na:1.8.0_77] at java.io.BufferedReader.fill(BufferedReader.java:161) ~[na:1.8.0_77] at java.io.BufferedReader.read(BufferedReader.java:182) ~[na:1.8.0_77] at org.apache.commons.net.io.CRLFLineReader.readLine(CRLFLineReader.java:58) ~[commons-net-3.3.jar:3.3] at org.apache.commons.net.ftp.FTP.__getReply(FTP.java:314) ~[commons-net-3.3.jar:3.3] at org.apache.commons.net.ftp.FTP.__getReply(FTP.java:294) ~[commons-net-3.3.jar:3.3] at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:483) ~[commons-net-3.3.jar:3.3] at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:608) ~[commons-net-3.3.jar:3.3] at org.apache.commons.net.ftp.FTP.cwd(FTP.java:828) ~[commons-net-3.3.jar:3.3] at org.apache.commons.net.ftp.FTPClient.changeWorkingDirectory(FTPClient.java:1128) ~[commons-net-3.3.jar:3.3] at org.apache.nifi.processors.standard.util.FTPTransfer.resetWorkingDirectory(FTPTransfer.java:360) ~[nifi-standard-processors-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10] at org.apache.nifi.processors.standard.util.FTPTransfer.getClient(FTPTransfer.java:494) ~[nifi-standard-processors-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10] at org.apache.nifi.processors.standard.util.FTPTransfer.flush(FTPTransfer.java:292) ~[nifi-standard-processors-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10] at org.apache.nifi.processors.standard.FetchFileTransfer.onTrigger(FetchFileTransfer.java:244) ~[nifi-standard-processors-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10] at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) [nifi-api-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10] at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099) [nifi-framework-core-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10] at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) [nifi-framework-core-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_77] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_77] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_77] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_77] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_77] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_77] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77] 2017-06-13 11:24:20,646 ERROR [Timer-Driven Process Thread-5] o.a.nifi.processors.standard.FetchFTP FetchFTP[id=0b272963-015c-1000-b879-09a07fe75263] Failed to fetch content for StandardFlowFileRecord[uuid=eeb860a8-b3fb-4c9c-804c-a4560aed1d4e,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1497338630570-3, container=default, section=3], offset=0, length=2097717],offset=0,name=TIAGPRSVMPT063245,size=2097717] from filename /edb4tel/data/DMP/TIAGPRSVMPT063245 on remote host 172.30.129.1:21 due to java.net.SocketTimeoutException: Read timed out; routing to failure: java.net.SocketTimeoutException: Read timed out 2017-06-13 11:24:20,647 ERROR

[#5]

Thanks for sharing updates. I didn't find anything suspicious in bootstrap log. The error message indicates there are some network issue between NiFi node and FTP server.

filename /edb4tel/data/DMP/TIAGPRSVMPT063246 on remote host 172.30.129.1:21 due to java.net.SocketTimeoutException: Read timed out

Can you confirm NiFi node can access to the FTP server port 21? Using telnet command, you can check connectivity:

$ telnet 172.30.129.1 21
(If successfully connected, the response from FTP server will be shown)

[#6]

Nifi node can access to FTP server port. Telnet, ftp-command line..... all ok....

[#7]

Thanks for trying. I haven't been able to reproduce the Read timed out issue. ListFTP/SFTP, GetFTP ... etc just work fine in my environment. I confirmed that it can download large file about 1GB.

There's another user facing the same issue, though..

https://community.hortonworks.com/questions/107715/nifi-130-getftp-read-timed-out.html

Did you try other FTP servers, just to confirm if the issue is FTP server specific. For example, a public FTP server such as the one to be used to download apache binaries, http://www.apache.org/dyn/closer.cgi/httpd/binaries/

If those processors work with other FTP servers, then I'd suggest looking at the FTP server's log as well.

[#8]

ListFTP-processor gives us list of files from remote srver and then FetchFTP returns "Read timed out" error

[#9]

Also I can't stop processor...

"do you see a number (usually 1 or 2) at the right top corner of the processor keeps being shown?" - yes, I see number)

[#10]

Hello, I am facing the same issue. ListFTP works fine but FetchFTP gives me Read Timed Out exception. Did you find a solution? I would be very appreciated.

Regards