Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How can I write to remote HDFS using NiFi?( Assume NiFi and HDP running on separate nodes or HDFS in cloud)

Solved Go to solution
Highlighted

How can I write to remote HDFS using NiFi?( Assume NiFi and HDP running on separate nodes or HDFS in cloud)

 
1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: How can I write to remote HDFS using NiFi?( Assume NiFi and HDP running on separate nodes or HDFS in cloud)

Super Guru

@milind pandit If i understand your ? correct you simply want to write to a HDFS. The HDFS cluster is running on different nodes which is typical. Use the putHDFS processor (info here). In the hadoop configuration resources simply point to core-site.xml and hdfs-site.xml. You can download these files from your target cluster. Now NiFi knows where the cluster is.

View solution in original post

5 REPLIES 5
Highlighted

Re: How can I write to remote HDFS using NiFi?( Assume NiFi and HDP running on separate nodes or HDFS in cloud)

Hi @milind pandit,

Yes you just need to have core-site and hdfs-site configuration files available on your NiFi nodes for your PutHDFS processor. And obviously that network connections are OK between NiFi and HDP.

Hope this helps.

Highlighted

Re: How can I write to remote HDFS using NiFi?( Assume NiFi and HDP running on separate nodes or HDFS in cloud)

Super Guru

@milind pandit If i understand your ? correct you simply want to write to a HDFS. The HDFS cluster is running on different nodes which is typical. Use the putHDFS processor (info here). In the hadoop configuration resources simply point to core-site.xml and hdfs-site.xml. You can download these files from your target cluster. Now NiFi knows where the cluster is.

View solution in original post

Highlighted

Re: How can I write to remote HDFS using NiFi?( Assume NiFi and HDP running on separate nodes or HDFS in cloud)

Super Guru

@milind pandit if you are running nodes in the cloud make sure the ports are accessible by both clusters. You will need to allow both clusters to talk with each other.

Re: How can I write to remote HDFS using NiFi?( Assume NiFi and HDP running on separate nodes or HDFS in cloud)

Thanks Sunile, the scenario is HDP in cloud and HDF on prem. Let me give a try.

Highlighted

Re: How can I write to remote HDFS using NiFi?( Assume NiFi and HDP running on separate nodes or HDFS in cloud)

Super Guru

@milind pandit that is straight forward. Simple download the two files from the cluster and place in your nifi cluster.

Don't have an account?
Coming from Hortonworks? Activate your account here