Skip to main content
totti1
totti1
Silver ⭐️⭐️
March 1, 2021
Solved

How to connect apache nifi with hadoop?

hello guys hope you doing well I want to put data into hdfs using putHDFS processor of apache nifi and keep getting me invalid processor here's my configuration conf and here's my pipeline pipeline Hope there's a good tutorial that can help me ? Thank you in advance !!

Shelton
SheltonAnswer
Gold ⭐️⭐️⭐️
March 1, 2021

@totti1

Nifi cluste is not aware of your Hadoop cluster until you copy these 2 files from your cluster /etc/hadoop/conf/hdfs-site.xml or /etc/hadoop/conf/core-site.xml to your local nifi installation

Hadoop configuration resources=/local/dir/hdfs-site.xml,/local/dir/core-site.xml
look for any of these processor group for HDFSprocessor.JPG

Hope that helps

View in thread

6 replies

Shelton
SheltonAnswer
Gold ⭐️⭐️⭐️
March 1, 2021

@totti1

Nifi cluste is not aware of your Hadoop cluster until you copy these 2 files from your cluster /etc/hadoop/conf/hdfs-site.xml or /etc/hadoop/conf/core-site.xml to your local nifi installation

Hadoop configuration resources=/local/dir/hdfs-site.xml,/local/dir/core-site.xml
look for any of these processor group for HDFSprocessor.JPG

Hope that helps

Translate
totti1
totti1Author
Silver ⭐️⭐️
March 1, 2021

Thank you for the reply 

first of all I am installing apache nifi in windows and cloudera in a virtual machine

how to put those files in putHDFS processor configuration ???

Translate
totti1
totti1Author
Silver ⭐️⭐️
March 1, 2021

And now I'm getting this error

  • 20:57:07 CET WARNING
    PutHDFS[id=ecd594bf-0177-1000-916a-aeb3f6dc26ca] Error stopping FileSystem statistics thread: null
     
  • 20:57:07 CET ERROR
    PutHDFS[id=ecd594bf-0177-1000-916a-aeb3f6dc26ca] Failed to properly initialize Processor. If still scheduled to run, NiFi will attempt to initialize and run the Processor again after the 'Administrative Yield Duration' has elapsed. Failure is due to java.nio.channels.UnresolvedAddressException: java.nio.channels.UnresolvedAddressException
     
  • 20:57:07 CET ERROR
    PutHDFS[id=ecd594bf-0177-1000-916a-aeb3f6dc26ca] HDFS Configuration error - null: java.nio.channels.UnresolvedAddressException
     
  • 20:56:35 CET WARNING
    PutHDFS[id=ecd594bf-0177-1000-916a-aeb3f6dc26ca] Error stopping FileSystem statistics thread: null
     
  • 20:56:35 CET ERROR
    PutHDFS[id=ecd594bf-0177-1000-916a-aeb3f6dc26ca] Failed to properly initialize Processor. If still scheduled to run, NiFi will attempt to initialize and run the Processor again after the 'Administrative Yield Duration' has elapsed. Failure is due to java.nio.channels.UnresolvedAddressException: java.nio.channels.UnresolvedAddressException
     
Translate
Terms & ConditionsAccessibility statement