Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How to connect apache nifi with hadoop?

avatar
Explorer

hello guys hope you doing well I want to put data into hdfs using putHDFS processor of apache nifi and keep getting me invalid processor here's my configuration conf and here's my pipeline pipeline Hope there's a good tutorial that can help me ? Thank you in advance !!

2 ACCEPTED SOLUTIONS

avatar
Master Mentor

@totti1

Nifi cluste is not aware of your Hadoop cluster until you copy these 2 files from your cluster /etc/hadoop/conf/hdfs-site.xml or /etc/hadoop/conf/core-site.xml to your local nifi installation

Hadoop configuration resources=/local/dir/hdfs-site.xml,/local/dir/core-site.xml
look for any of these processor group for HDFSprocessor.JPG

Hope that helps

View solution in original post

avatar
Master Mentor

@totti1 

You will need to copy the hdfs/core-site.xml to a local path accessible to your windows. And you will need to update your host's file entry to make the VM reachable from the windows machine. 

You should be able to ping your vm from the windows machine and vice versa.

Edit and change core-site.xml and hdfs-site.xml files and remove the FQDN:8020 to an IP ie for class C network like 192.168.10.201:8020 restart the processors and let me know.

Hope that helps?

View solution in original post

6 REPLIES 6

avatar
Master Mentor

@totti1

Nifi cluste is not aware of your Hadoop cluster until you copy these 2 files from your cluster /etc/hadoop/conf/hdfs-site.xml or /etc/hadoop/conf/core-site.xml to your local nifi installation

Hadoop configuration resources=/local/dir/hdfs-site.xml,/local/dir/core-site.xml
look for any of these processor group for HDFSprocessor.JPG

Hope that helps

avatar
Explorer

Thank you for the reply 

first of all I am installing apache nifi in windows and cloudera in a virtual machine

how to put those files in putHDFS processor configuration ???

avatar
Explorer

And now I'm getting this error

  • 20:57:07 CET WARNING
    PutHDFS[id=ecd594bf-0177-1000-916a-aeb3f6dc26ca] Error stopping FileSystem statistics thread: null
     
  • 20:57:07 CET ERROR
    PutHDFS[id=ecd594bf-0177-1000-916a-aeb3f6dc26ca] Failed to properly initialize Processor. If still scheduled to run, NiFi will attempt to initialize and run the Processor again after the 'Administrative Yield Duration' has elapsed. Failure is due to java.nio.channels.UnresolvedAddressException: java.nio.channels.UnresolvedAddressException
     
  • 20:57:07 CET ERROR
    PutHDFS[id=ecd594bf-0177-1000-916a-aeb3f6dc26ca] HDFS Configuration error - null: java.nio.channels.UnresolvedAddressException
     
  • 20:56:35 CET WARNING
    PutHDFS[id=ecd594bf-0177-1000-916a-aeb3f6dc26ca] Error stopping FileSystem statistics thread: null
     
  • 20:56:35 CET ERROR
    PutHDFS[id=ecd594bf-0177-1000-916a-aeb3f6dc26ca] Failed to properly initialize Processor. If still scheduled to run, NiFi will attempt to initialize and run the Processor again after the 'Administrative Yield Duration' has elapsed. Failure is due to java.nio.channels.UnresolvedAddressException: java.nio.channels.UnresolvedAddressException
     

avatar
Master Mentor

@totti1 

You will need to copy the hdfs/core-site.xml to a local path accessible to your windows. And you will need to update your host's file entry to make the VM reachable from the windows machine. 

You should be able to ping your vm from the windows machine and vice versa.

Edit and change core-site.xml and hdfs-site.xml files and remove the FQDN:8020 to an IP ie for class C network like 192.168.10.201:8020 restart the processors and let me know.

Hope that helps?

avatar
Explorer

It worked Thank you so much

avatar
Moderator

For future reference adding here the link to our public documentation on how to connect NiFi with Hive on CDP.


Ferenc Erdelyi, Technical Solutions Manager

Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community: