How to ingest files into HDFS via Apache NiFi from non-hadoop environment

[#1]

@Matt Clarke, @Timothy Spann, @Shishir Saxena, @Bryan Bende, @milind pandit, @Dan Chaffelson, @Pierre Villard, @Andrew Grande

Apache NiFi is installed on non-hadoop environment and targets to ingest processed files into HDFS (Kerberized cluster).

Is it workable solution to achieve above use case as I face multiple error even after performing below activities. Please advise if there is anything additional I have to perform.

* Firewall restriction between NiFi and management server is open and ports (22,88,749,389) are open.

* Firewall restriction between NiFi and edge node server is open and ports (22, 2181,9083) are open

* krb5.conf file from hadoop cluster along with keytab for application user is copied to NiFi server. Running kinit using application user and keytab - successful token is listed under klist.

* SSH operation is successful and also SFTP into hadoop server works fine.

* configured hdfs-site.xml and core-site.xml files into NiFi.

* PutHDFS processor fails to ingest data throwing authentication error. Is there something I am missing here.

Hadoop environment : 4 management node and 1 edge node on public network within cluster and 4 worker nodes on private network.

Alternative solution to this, I have installed NiFi in edge node and everything works fine. I need to make Apache NiFi work on non-hadoop environment and ingest data into HDFS and Hive.

[#2]

@Mothilal marimuthu

What is the exact error being thrown?

What version of NiFi?

[#3]

Hi @Mothilal marimuthu,

You need also to open the port in the firewall for HDFS and their different subservice like namenode datanode etc... You have to do that for each service that your need

If you still have the default configuration here you have a list of port:

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.1/bk_reference/content/hdfs-ports.html

Michel

[#4]

Anything in the logs? Make sure that id running nifi has permissions to the files. Login as that user and try to read the files.

Test Kerberos client on the nifi box

Please post error images, the exported nifi template, configuration files and logs.

Any firewall or blockers running on the nifi node?

Can you write to HDFS from the command line?

Is the HDFS directory created? writable by the NiFi user? Permissions proper.

Try a 777 directory in HDFS first.

Check the HDFS directory