Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Nifi data streaming into HDFS

Solved Go to solution
Highlighted

Nifi data streaming into HDFS

Contributor

I want to stream data into HDFS. I have Nifi running on my laptop. I have files that I want to transfer to HDFS on HDP on AWS. what is the correct process?

I was trying to use the getFile process and the putHDFS to move it. But the PutHDFS I couldn't get to work.

I dont have Kerberos and how do I log into HDFS?

9922-screen-shot-2016-11-30-at-125457-am.png

9923-screen-shot-2016-11-30-at-125436-am.png

9924-screen-shot-2016-11-30-at-125414-am.png

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Nifi data streaming into HDFS

Guru

Your hdfs-site.xml will have the connection info to hdfs.

I believe the problem is your directory in the first screenshot -- you only need a hdfs path (and not the connection info hdfs://server).

Example:

9916-screen-shot-2016-11-30-at-73653-am.png

(But as @Avijeet Dash suggests, looking at exact error (either by clicking the processor error icon, or nifi-app.log for more info) is useful.

View solution in original post

7 REPLIES 7
Highlighted

Re: Nifi data streaming into HDFS

Expert Contributor

Hi @Dagmawi Mengistu

Can you click on the Error mark to see the exact error, also you can check nifi-app.log and post more details.

Highlighted

Re: Nifi data streaming into HDFS

Contributor

Hi @Avijeet Dash

The problem is I have Nifi only on my Laptop and not on my cluster, How does Nifi on my local mac gets access to the conf files? Here is the error...

9939-screen-shot-2016-11-30-at-95233-am.png

Highlighted

Re: Nifi data streaming into HDFS

Guru

See comment to answer above on how to get configs to local.

Highlighted

Re: Nifi data streaming into HDFS

Guru

Your hdfs-site.xml will have the connection info to hdfs.

I believe the problem is your directory in the first screenshot -- you only need a hdfs path (and not the connection info hdfs://server).

Example:

9916-screen-shot-2016-11-30-at-73653-am.png

(But as @Avijeet Dash suggests, looking at exact error (either by clicking the processor error icon, or nifi-app.log for more info) is useful.

View solution in original post

Re: Nifi data streaming into HDFS

Contributor

@Greg Keys I tried to change it but it didn't work. look at the comment I left above.

Highlighted

Re: Nifi data streaming into HDFS

Guru

@Dagmawi Mengistu To get the configs:

  1. login to your cluster via Ambari
  2. click the HDFS service on left
  3. in upper right, Service Actions dropdown, select Download Client Configs

This will download to your local machine, and when you unpack this you will find core-site.xml

Place core-site.xml anywhere locally and use this path in your PutHDFS config.

Highlighted

Re: Nifi data streaming into HDFS

Expert Contributor

The error means that the processor has not been configured completely yet. make sure you have auto terminated the success relation. for failure you can connect back to puthdfs.

Don't have an account?
Coming from Hortonworks? Activate your account here