question Re: Nifi data streaming into HDFS in Support Questions

Nifi data streaming into HDFS

doug_mengistu — Mon, 19 Aug 2019 09:41:08 GMT

I want to stream data into HDFS. I have Nifi running on my laptop. I have files that I want to transfer to HDFS on HDP on AWS. what is the correct process?

I was trying to use the getFile process and the putHDFS to move it. But the PutHDFS I couldn't get to work.

I dont have Kerberos and how do I log into HDFS?

Re: Nifi data streaming into HDFS

avijeetd — Wed, 30 Nov 2016 14:12:55 GMT

Hi @Dagmawi Mengistu

Can you click on the Error mark to see the exact error, also you can check nifi-app.log and post more details.

Re: Nifi data streaming into HDFS

gkeys — Mon, 19 Aug 2019 09:40:49 GMT

Your hdfs-site.xml will have the connection info to hdfs.

I believe the problem is your directory in the first screenshot -- you only need a hdfs path (and not the connection info hdfs://server).

Example:

(But as @Avijeet Dash suggests, looking at exact error (either by clicking the processor error icon, or nifi-app.log for more info) is useful.

Re: Nifi data streaming into HDFS

knarayanan — Wed, 30 Nov 2016 22:19:59 GMT

The error means that the processor has not been configured completely yet. make sure you have auto terminated the success relation. for failure you can connect back to puthdfs.

Re: Nifi data streaming into HDFS

doug_mengistu — Wed, 30 Nov 2016 23:08:37 GMT

@Greg Keys I tried to change it but it didn't work. look at the comment I left above.

Re: Nifi data streaming into HDFS

doug_mengistu — Mon, 19 Aug 2019 09:40:41 GMT

Hi @Avijeet Dash

The problem is I have Nifi only on my Laptop and not on my cluster, How does Nifi on my local mac gets access to the conf files? Here is the error...

Re: Nifi data streaming into HDFS

gkeys — Wed, 30 Nov 2016 23:24:51 GMT

@Dagmawi Mengistu To get the configs:

login to your cluster via Ambari
click the HDFS service on left
in upper right, Service Actions dropdown, select Download Client Configs

This will download to your local machine, and when you unpack this you will find core-site.xml

Place core-site.xml anywhere locally and use this path in your PutHDFS config.

Re: Nifi data streaming into HDFS

gkeys — Wed, 30 Nov 2016 23:26:14 GMT

See comment to answer above on how to get configs to local.