I have HDF cluster with 3 Nifi instance which lunches jobs(Hive/Spark) on HDP cluster. Usually nifi writes all information to different repositories available on local machine.
My question is - Does nifi writes any data,provenance information or does spilling on HDP nodes (ex. data nodes in HDP cluster) while accessing HDFS,Hive or Spark services ?
Hello Shashi, NiFi write in its 3 repositories (flow files, content and provenance) on local nodes only; you can certainly do provenance export in Atlas for example (there's some work around that) but it's not embedded.
Thanks Laurent. I agree on that. I am trying to get detailed understanding of communication between HDF and HDP cluster. When HDF Nifi connects (via HDFS processor , Hive connection or spark ) to HDP cluster, does it writes anything to local disks of data nodes of HDP ?