Support Questions

Find answers, ask questions, and share your expertise

Nifi data streaming into HDFS

Contributor

I want to stream data into HDFS. I have Nifi running on my laptop. I have files that I want to transfer to HDFS on HDP on AWS. what is the correct process?

I was trying to use the getFile process and the putHDFS to move it. But the PutHDFS I couldn't get to work.

I dont have Kerberos and how do I log into HDFS?

9922-screen-shot-2016-11-30-at-125457-am.png

9923-screen-shot-2016-11-30-at-125436-am.png

9924-screen-shot-2016-11-30-at-125414-am.png

1 ACCEPTED SOLUTION

Guru

Your hdfs-site.xml will have the connection info to hdfs.

I believe the problem is your directory in the first screenshot -- you only need a hdfs path (and not the connection info hdfs://server).

Example:

9916-screen-shot-2016-11-30-at-73653-am.png

(But as @Avijeet Dash suggests, looking at exact error (either by clicking the processor error icon, or nifi-app.log for more info) is useful.

View solution in original post

7 REPLIES 7

Expert Contributor

Hi @Dagmawi Mengistu

Can you click on the Error mark to see the exact error, also you can check nifi-app.log and post more details.

Contributor

Hi @Avijeet Dash

The problem is I have Nifi only on my Laptop and not on my cluster, How does Nifi on my local mac gets access to the conf files? Here is the error...

9939-screen-shot-2016-11-30-at-95233-am.png

Guru

See comment to answer above on how to get configs to local.

Guru

Your hdfs-site.xml will have the connection info to hdfs.

I believe the problem is your directory in the first screenshot -- you only need a hdfs path (and not the connection info hdfs://server).

Example:

9916-screen-shot-2016-11-30-at-73653-am.png

(But as @Avijeet Dash suggests, looking at exact error (either by clicking the processor error icon, or nifi-app.log for more info) is useful.

Contributor

@Greg Keys I tried to change it but it didn't work. look at the comment I left above.

Guru

@Dagmawi Mengistu To get the configs:

  1. login to your cluster via Ambari
  2. click the HDFS service on left
  3. in upper right, Service Actions dropdown, select Download Client Configs

This will download to your local machine, and when you unpack this you will find core-site.xml

Place core-site.xml anywhere locally and use this path in your PutHDFS config.

Expert Contributor

The error means that the processor has not been configured completely yet. make sure you have auto terminated the success relation. for failure you can connect back to puthdfs.

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.