question Re: How to connect apache nifi with hadoop? in Support Questions

How to connect apache nifi with hadoop?

totti1 — Mon, 01 Mar 2021 11:58:55 GMT

hello guys hope you doing well I want to put data into hdfs using putHDFS processor of apache nifi and keep getting me invalid processor here's my configuration conf and here's my pipeline pipeline Hope there's a good tutorial that can help me ? Thank you in advance !!

Re: How to connect apache nifi with hadoop?

Shelton — Mon, 01 Mar 2021 18:11:16 GMT

@totti1

Nifi cluste is not aware of your Hadoop cluster until you copy these 2 files from your cluster /etc/hadoop/conf/hdfs-site.xml or /etc/hadoop/conf/core-site.xml to your local nifi installation

Hadoop configuration resources=/local/dir/hdfs-site.xml,/local/dir/core-site.xml
look for any of these processor group for HDFS

Hope that helps

Re: How to connect apache nifi with hadoop?

totti1 — Mon, 01 Mar 2021 19:49:22 GMT

Thank you for the reply

first of all I am installing apache nifi in windows and cloudera in a virtual machine

how to put those files in putHDFS processor configuration ???

Re: How to connect apache nifi with hadoop?

totti1 — Mon, 01 Mar 2021 20:01:13 GMT

And now I'm getting this error

20:57:07 CET WARNING
PutHDFS[id=ecd594bf-0177-1000-916a-aeb3f6dc26ca] Error stopping FileSystem statistics thread: null
20:57:07 CET ERROR
PutHDFS[id=ecd594bf-0177-1000-916a-aeb3f6dc26ca] Failed to properly initialize Processor. If still scheduled to run, NiFi will attempt to initialize and run the Processor again after the 'Administrative Yield Duration' has elapsed. Failure is due to java.nio.channels.UnresolvedAddressException: java.nio.channels.UnresolvedAddressException
20:57:07 CET ERROR
PutHDFS[id=ecd594bf-0177-1000-916a-aeb3f6dc26ca] HDFS Configuration error - null: java.nio.channels.UnresolvedAddressException
20:56:35 CET WARNING
PutHDFS[id=ecd594bf-0177-1000-916a-aeb3f6dc26ca] Error stopping FileSystem statistics thread: null
20:56:35 CET ERROR
PutHDFS[id=ecd594bf-0177-1000-916a-aeb3f6dc26ca] Failed to properly initialize Processor. If still scheduled to run, NiFi will attempt to initialize and run the Processor again after the 'Administrative Yield Duration' has elapsed. Failure is due to java.nio.channels.UnresolvedAddressException: java.nio.channels.UnresolvedAddressException

Re: How to connect apache nifi with hadoop?

Shelton — Mon, 01 Mar 2021 20:41:20 GMT

@totti1

You will need to copy the hdfs/core-site.xml to a local path accessible to your windows. And you will need to update your host's file entry to make the VM reachable from the windows machine.

You should be able to ping your vm from the windows machine and vice versa.

Edit and change core-site.xml and hdfs-site.xml files and remove the FQDN:8020 to an IP ie for class C network like 192.168.10.201:8020 restart the processors and let me know.

Hope that helps?

Re: How to connect apache nifi with hadoop?

totti1 — Tue, 02 Mar 2021 10:45:55 GMT

It worked Thank you so much

Re: How to connect apache nifi with hadoop?

Bender — Tue, 23 Mar 2021 14:50:57 GMT

For future reference adding here the link to our public documentation on how to connect NiFi with Hive on CDP.