How to connect apache nifi with hadoop?

[#1]

hello guys hope you doing well I want to put data into hdfs using putHDFS processor of apache nifi and keep getting me invalid processor here's my configuration conf and here's my pipeline pipeline Hope there's a good tutorial that can help me ? Thank you in advance !!

[#2 · Accepted solution]

@totti1

Nifi cluste is not aware of your Hadoop cluster until you copy these 2 files from your cluster /etc/hadoop/conf/hdfs-site.xml or /etc/hadoop/conf/core-site.xml to your local nifi installation

Hadoop configuration resources=/local/dir/hdfs-site.xml,/local/dir/core-site.xml
look for any of these processor group for HDFS

Hope that helps

[#3 · Accepted solution]

@totti1

Nifi cluste is not aware of your Hadoop cluster until you copy these 2 files from your cluster /etc/hadoop/conf/hdfs-site.xml or /etc/hadoop/conf/core-site.xml to your local nifi installation

Hadoop configuration resources=/local/dir/hdfs-site.xml,/local/dir/core-site.xml
look for any of these processor group for HDFS

Hope that helps

[#4]

Thank you for the reply 

first of all I am installing apache nifi in windows and cloudera in a virtual machine

how to put those files in putHDFS processor configuration ???

[#5]

And now I'm getting this error

• 20:57:07 CET WARNING
  PutHDFS[id=ecd594bf-0177-1000-916a-aeb3f6dc26ca] Error stopping FileSystem statistics thread: null
   
• 20:57:07 CET ERROR
  PutHDFS[id=ecd594bf-0177-1000-916a-aeb3f6dc26ca] Failed to properly initialize Processor. If still scheduled to run, NiFi will attempt to initialize and run the Processor again after the 'Administrative Yield Duration' has elapsed. Failure is due to java.nio.channels.UnresolvedAddressException: java.nio.channels.UnresolvedAddressException
   
• 20:57:07 CET ERROR
  PutHDFS[id=ecd594bf-0177-1000-916a-aeb3f6dc26ca] HDFS Configuration error - null: java.nio.channels.UnresolvedAddressException
   
• 20:56:35 CET WARNING
  PutHDFS[id=ecd594bf-0177-1000-916a-aeb3f6dc26ca] Error stopping FileSystem statistics thread: null
   
• 20:56:35 CET ERROR
  PutHDFS[id=ecd594bf-0177-1000-916a-aeb3f6dc26ca] Failed to properly initialize Processor. If still scheduled to run, NiFi will attempt to initialize and run the Processor again after the 'Administrative Yield Duration' has elapsed. Failure is due to java.nio.channels.UnresolvedAddressException: java.nio.channels.UnresolvedAddressException
   

[#6 · Accepted solution]

@totti1 

You will need to copy the hdfs/core-site.xml to a local path accessible to your windows. And you will need to update your host's file entry to make the VM reachable from the windows machine. 

You should be able to ping your vm from the windows machine and vice versa.

Edit and change core-site.xml and hdfs-site.xml files and remove the FQDN:8020 to an IP ie for class C network like 192.168.10.201:8020 restart the processors and let me know.

Hope that helps?

[#7]

It worked Thank you so much

[#8]

For future reference adding here the link to our public documentation on how to connect NiFi with Hive on CDP.