Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to copy data From hdfs to Nifi(Nifi is in separate Cluster and Hadoop is in separate cluster)

Solved Go to solution
Highlighted

How to copy data From hdfs to Nifi(Nifi is in separate Cluster and Hadoop is in separate cluster)

Explorer

Hi All,

i have created 2 node nifi cluster, and 3 node hadoop cluster(hortonwork), Now i want to interact with hdfs from Nifi for this what are the configuration properties i have to set. is have to copy core-site.xml into nifi bin? , then how to create processor for copy data from one hdfs directory to another . please explain with simple example.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: How to copy data From hdfs to Nifi(Nifi is in separate Cluster and Hadoop is in separate cluster)

You can use PutHDFS or FetchHDFS depending on your using. You need to set the properties Hadoop Configuration Resources in the processor which you can point to the location where you have core-site and hdfs-site configuration files. Make sure network connections are OK between NiFi and HDP. Also didnt understand your usecase fully, "Why you would like to use NiFI to copy files between HDFS directories?"

View solution in original post

4 REPLIES 4
Highlighted

Re: How to copy data From hdfs to Nifi(Nifi is in separate Cluster and Hadoop is in separate cluster)

Rising Star

Hi @AnjiReddy Anumolu I would take a look at the twitter streaming example. It puts files into HDFS but the configuration is somewhat similar.

https://community.hortonworks.com/articles/1282/sample-hdfnifi-flow-to-push-tweets-into-solrbanana.h...

Re: How to copy data From hdfs to Nifi(Nifi is in separate Cluster and Hadoop is in separate cluster)

You can use PutHDFS or FetchHDFS depending on your using. You need to set the properties Hadoop Configuration Resources in the processor which you can point to the location where you have core-site and hdfs-site configuration files. Make sure network connections are OK between NiFi and HDP. Also didnt understand your usecase fully, "Why you would like to use NiFI to copy files between HDFS directories?"

View solution in original post

Highlighted

Re: How to copy data From hdfs to Nifi(Nifi is in separate Cluster and Hadoop is in separate cluster)

Super Guru

Are you looking to use NiFi (on a different cluster) to move files within the Hadoop cluster? This may not be the most efficient approach as you will be moving files off the cluster just to move them back again.

If you can install NiFi on the Hadoop cluster (or so it has a Hadoop client), you could use ExecuteProcess or ExecuteStreamCommand to do something like "hadoop fs -copy /path/to/file /new/path/to/file".

Highlighted

Re: How to copy data From hdfs to Nifi(Nifi is in separate Cluster and Hadoop is in separate cluster)

Explorer

Hi @milind pandit, @AnjiReddy Anumolu,

I am facing the same issue. Would you please share how did you resolve this issue? I am confused about the GetFile and PutHDFS processor configuration. let me explain a little more:

I just want to setup a data flow with nifi for now to test whether i can transfer files from HDF cluster to HDFS of HDP cluster.For that I am just using two processor "GetFile" and "PutHDFS". I deployed 2 clusters with 2 nodes of each- HDF 3.1(2 nodes-Nifi and services) and HDP 2.6.4(2 nodes- Master and worker). Now I want to transfer files through Nifi and write those data in HDFS . How can I do that? Please share your experiences how you resolved this. Thank you.

Don't have an account?
Coming from Hortonworks? Activate your account here