Nifi data streaming into HDFS

[#1]

I want to stream data into HDFS. I have Nifi running on my laptop. I have files that I want to transfer to HDFS on HDP on AWS. what is the correct process?

I was trying to use the getFile process and the putHDFS to move it. But the PutHDFS I couldn't get to work.

I dont have Kerberos and how do I log into HDFS?

[#2]

Hi @Dagmawi Mengistu

Can you click on the Error mark to see the exact error, also you can check nifi-app.log and post more details.

[#3]

Hi @Avijeet Dash

The problem is I have Nifi only on my Laptop and not on my cluster, How does Nifi on my local mac gets access to the conf files? Here is the error...

[#4]

See comment to answer above on how to get configs to local.

[#5 · Accepted solution]

Your hdfs-site.xml will have the connection info to hdfs.

I believe the problem is your directory in the first screenshot -- you only need a hdfs path (and not the connection info hdfs://server).

Example:

(But as @Avijeet Dash suggests, looking at exact error (either by clicking the processor error icon, or nifi-app.log for more info) is useful.

[#6]

@Greg Keys I tried to change it but it didn't work. look at the comment I left above.

[#7]

@Dagmawi Mengistu To get the configs:

1. login to your cluster via Ambari
2. click the HDFS service on left
3. in upper right, Service Actions dropdown, select Download Client Configs

This will download to your local machine, and when you unpack this you will find core-site.xml

Place core-site.xml anywhere locally and use this path in your PutHDFS config.

[#8]

The error means that the processor has not been configured completely yet. make sure you have auto terminated the success relation. for failure you can connect back to puthdfs.