Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HDF NIfi - Does Nifi writes provenance/data on HDP Node ?

HDF NIfi - Does Nifi writes provenance/data on HDP Node ?

Contributor

Hi

I have HDF cluster with 3 Nifi instance which lunches jobs(Hive/Spark) on HDP cluster. Usually nifi writes all information to different repositories available on local machine.

My question is - Does nifi writes any data,provenance information or does spilling on HDP nodes (ex. data nodes in HDP cluster) while accessing HDFS,Hive or Spark services ?

Thanks

3 REPLIES 3

Re: HDF NIfi - Does Nifi writes provenance/data on HDP Node ?

Guru

Hello Shashi, NiFi write in its 3 repositories (flow files, content and provenance) on local nodes only; you can certainly do provenance export in Atlas for example (there's some work around that) but it's not embedded.

Re: HDF NIfi - Does Nifi writes provenance/data on HDP Node ?

Contributor

Thanks Laurent. I agree on that. I am trying to get detailed understanding of communication between HDF and HDP cluster. When HDF Nifi connects (via HDFS processor , Hive connection or spark ) to HDP cluster, does it writes anything to local disks of data nodes of HDP ?

Re: HDF NIfi - Does Nifi writes provenance/data on HDP Node ?

Guru

no it won't, it's acting like a regular HDFS client (well, it is)