Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

NiFi: Hadoop Configuration Error

avatar

We have integrated NiFi service (HDF service) to our existing HDP cluster. NiFi installation was successful and NiFi is running without any error. But there is an error with PutHDFS. It's showing that there is an error in our Hadoop configuration. I'm posting the error and screenshot too.

2018-06-20 12:00:14,246 WARN [StandardProcessScheduler Thread-6] org.apache.hadoop.conf.Configuration /tmp/core-site.xml:an attempt to override final parameter: fs.defaultFS;  Ignoring.
2018-06-20 12:00:14,248 ERROR [StandardProcessScheduler Thread-6] o.apache.nifi.processors.hadoop.PutHDFS PutHDFS[id=11a40827-0164-1000-ffff-ffffb07a04d9] HDFS Configuration error - java.net.ConnectException: Connection refused: {}

77789-screenshot-10.png

77790-screenshot-5.png

java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
        at org.apache.nifi.processors.hadoop.AbstractHadoopProcessor.checkHdfsUriForTimeout(AbstractHadoopProcessor.java:345)
        at org.apache.nifi.processors.hadoop.AbstractHadoopProcessor.resetHDFSResources(AbstractHadoopProcessor.java:260)
        at org.apache.nifi.processors.hadoop.AbstractHadoopProcessor.abstractOnScheduled(AbstractHadoopProcessor.java:205)
        at sun.reflect.GeneratedMethodAccessor728.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:137)
        at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:125)
        at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:70)
        at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:47)
        at org.apache.nifi.controller.StandardProcessorNode$1.call(StandardProcessorNode.java:1334)
        at org.apache.nifi.controller.StandardProcessorNode$1.call(StandardProcessorNode.java:1330)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
2018-06-20 12:00:14,248 ERROR [StandardProcessScheduler Thread-6] org.apache.nifi.engine.FlowEngine A flow controller task execution stopped abnormally
java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:192)
        at org.apache.nifi.engine.FlowEngine.afterExecute(FlowEngine.java:100)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1150)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException: null
2 REPLIES 2

avatar

Hey @Abhinav Phutela !
Are you able to telnet the 50070 or other Datanode ports of those hosts (hdfs hosts)?
I made a test here, you can try to use as an example:

77794-screen-shot-2018-06-21-at-172138.png

Content of GetFile Processor

77795-screen-shot-2018-06-21-at-172133.png

Content of the PutHDFS

77796-screen-shot-2018-06-21-at-172354.png

My flow above will send the hdfs_nifi.txt on the Nifi host and send to HDFS host.

#First on the HDFS side, need to send the *-site.xml config from the hadoop-client/conf dir to Nifi host
[root@hdfs_node1 conf]$ scp core-site.xml hdfs-site.xml root@nifi_node1:/home/nifi
#Then I'll spread the *-site.xml config to my other nifi instances and create the file to put on HDFS
[nifi@nifi_node1 ~]$ echo "testing hdfs content" > hdfs_nifi.txt [nifi@nifi_node1 ~]$ sudo chown nifi:hadoop *.xml [nifi@nifi_node1 ~]$ scp core-site.xml hdfs-site.xml nifi@nifi_node2:/home/nifi [nifi@nifi_node1 ~]$ scp core-site.xml hdfs-site.xml nifi@nifi_node3:/home/nifi

#After i start the job, got the succesfully data on HDFS
[root@hdfs_node1 conf]$ hdfs dfs -cat /tmp/hdfs_nifi.txt testing hdfs content

Ps: I also had changed my /etc/hosts to known the hosts from HDFS cluster.

Hope this helps!

avatar

What should be the permission of the hdfs-site.xml and core-site.xml files which I'm supposed to copy in /home/nifi folder?

And I'm able to telnet to datanode and namenode both (50075 and 50070)