question Re: problem with getFTP to putHDFS? in Archives of Support Questions (Read Only)

problem with getFTP to putHDFS?

iyappan — Tue, 31 May 2016 11:04:52 GMT

i try to move files from FTP directory to HDFS but shows error in image issue.

i'm using windows nifi so in core-sit.xml and hdfs-site.xml are stored in to the local windows machine and directly give target directory as /tmp/target in HDFS

The files are queued from FTP and shows error in HDFS

Thanks in advance

Re: problem with getFTP to putHDFS?

iyappan — Mon, 19 Aug 2019 11:32:40 GMT

I have enclosed a copy of the flow, which shows the error. Can anyone help me out for this.

Re: problem with getFTP to putHDFS?

pvillard — Tue, 31 May 2016 13:18:20 GMT

Could you please provide logs in <NIFI dir>/logs/nifi-app.log?

Re: problem with getFTP to putHDFS?

iyappan — Tue, 31 May 2016 13:53:56 GMT

2016-05-31 12:23:02,353 ERROR [StandardProcessScheduler Thread-1] o.a.n.controller.StandardProcessorNode Failed to invoke @OnScheduled method due to java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task. java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task. at org.apache.nifi.controller.StandardProcessorNode.invokeTaskAsCancelableFuture(StandardProcessorNode.java:1405) ~[na:na] at org.apache.nifi.controller.StandardProcessorNode.access$100(StandardProcessorNode.java:89) ~[na:na] at org.apache.nifi.controller.StandardProcessorNode$1.run(StandardProcessorNode.java:1243) ~[na:na] at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [na:1.8.0_91] at java.util.concurrent.FutureTask.run(Unknown Source) [na:1.8.0_91] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source) [na:1.8.0_91] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [na:1.8.0_91] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [na:1.8.0_91] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.8.0_91] at java.lang.Thread.run(Unknown Source) [na:1.8.0_91] Caused by: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException at java.util.concurrent.FutureTask.report(Unknown Source) [na:1.8.0_91] at java.util.concurrent.FutureTask.get(Unknown Source) [na:1.8.0_91] at org.apache.nifi.controller.StandardProcessorNode.invokeTaskAsCancelableFuture(StandardProcessorNode.java:1388) ~[na:na] ... 9 common frames omitted Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_91] at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[na:1.8.0_91] at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[na:1.8.0_91] at java.lang.reflect.Method.invoke(Unknown Source) ~[na:1.8.0_91] at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:137) ~[na:na] at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:125) ~[na:na] at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:70) ~[na:na] at org.apache.nifi.controller.StandardProcessorNode$1$1.call(StandardProcessorNode.java:1247) ~[na:na] at org.apache.nifi.controller.StandardProcessorNode$1$1.call(StandardProcessorNode.java:1243) ~[na:na] ... 6 common frames omitted Caused by: java.nio.channels.UnresolvedAddressException: null at sun.nio.ch.Net.checkAddress(Unknown Source) ~[na:1.8.0_91] at sun.nio.ch.SocketChannelImpl.connect(Unknown Source) ~[na:1.8.0_91] at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192) ~[na:na] at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530) ~[na:na] at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:494) ~[na:na] at org.apache.nifi.processors.hadoop.AbstractHadoopProcessor.checkHdfsUriForTimeout(AbstractHadoopProcessor.java:328) ~[na:na] at org.apache.nifi.processors.hadoop.AbstractHadoopProcessor.resetHDFSResources(AbstractHadoopProcessor.java:253) ~[na:na] at org.apache.nifi.processors.hadoop.AbstractHadoopProcessor.abstractOnScheduled(AbstractHadoopProcessor.java:196) ~[na:na] at org.apache.nifi.processors.hadoop.PutHDFS.onScheduled(PutHDFS.java:177) ~[na:na] ... 15 common frames omitted

Re: problem with getFTP to putHDFS?

pvillard — Tue, 31 May 2016 13:59:33 GMT

Looking at:

java.nio.channels.UnresolvedAddressException: null at sun.nio.ch.Net.checkAddress(Unknown Source) ~[na:1.8.0_91] at sun.nio.ch.SocketChannelImpl.connect(Unknown Source) ~[na:1.8.0_91] at

I'd say that there is a network issue between your NiFi instance and your HDFS cluster. Did you correctly configure the processor with IP addresses and ports? Are you sure it is reachable from your NiFi instance?

Re: problem with getFTP to putHDFS?

iyappan — Tue, 31 May 2016 14:09:41 GMT

how to check this?

Re: problem with getFTP to putHDFS?

iyappan — Tue, 31 May 2016 14:19:11 GMT

As you said earlier i download files core-site.xml,hdfs-site.xml from ambari placed in the windows machine

and placed the path in "Hadoop Configuration Resources" and directory as "/tmp/dest/".

it shows error like this

Re: problem with getFTP to putHDFS?

pvillard — Tue, 31 May 2016 14:48:54 GMT

core-site and hdfs-site contain information regarding IP addresses to use to connect your cluster. Depending of what is in your configuration files, your NiFi instance may not be able to reach HDFS.

Re: problem with getFTP to putHDFS?

iyappan — Tue, 31 May 2016 16:12:21 GMT

i tried to change core-site.xml and hdfs-site.xml files sandbox.hortonworks.com:8020 to 192.168.1.158:8020

its working fine