@Matt Clarke, @Timothy Spann, @Shishir Saxena, @Bryan Bende, @milind pandit, @Dan Chaffelson, @Pierre Villard, @Andrew Grande
Apache NiFi is installed on non-hadoop environment and targets to ingest processed files into HDFS (Kerberized cluster).
Is it workable solution to achieve above use case as I face multiple error even after performing below activities. Please advise if there is anything additional I have to perform.
* Firewall restriction between NiFi and management server is open and ports (22,88,749,389) are open.
* Firewall restriction between NiFi and edge node server is open and ports (22, 2181,9083) are open
* krb5.conf file from hadoop cluster along with keytab for application user is copied to NiFi server. Running kinit using application user and keytab - successful token is listed under klist.
* SSH operation is successful and also SFTP into hadoop server works fine.
* configured hdfs-site.xml and core-site.xml files into NiFi.
* PutHDFS processor fails to ingest data throwing authentication error. Is there something I am missing here.
Hadoop environment : 4 management node and 1 edge node on public network within cluster and 4 worker nodes on private network.
Alternative solution to this, I have installed NiFi in edge node and everything works fine. I need to make Apache NiFi work on non-hadoop environment and ingest data into HDFS and Hive.