Using HDFS processors without additional configuration in NiFi

[#1]

Hey there,

I'm currently trying to read files from the HDFS in the HDP cluster where NiFi is running (with HDF management pack). Is there a way that HDFS processors automatically get the configuration for core-site, hdfs-site and kerberos, without having to configure them every time I'm using the HDFS processors?

Best regards,

Markus

[#2]

@Markus Wilhelm

-

The documentation states the following:

Based on above you should be able to place your core-site.xml and hdfs.site.xml in the NiFi classpath. This will allow all HDFS based processors that do not have a configuration defined for this property to use those as the defaults.

-

Thank you,

Matt

-

When an "Answer" addresses/solves your question, please select "Accept" beneath that answer. This encourages user participation in this forum.

[#3 · Accepted solution]

@Markus Wilhelm

I don't think we can make NiFi to read kerberos configs to read by default but you can make use of Process group variables in your HDFS processor configs and define the variables scope as NiFi Flow so that you can use same variables across all the processors in NiFi instance.

You can copy hdfs-site.xml,core-site.xml to nifi lib path and restart nifi, then you don't have to specify the path because nifi will load all the .xml from lib path, but it's not recommended way of approach because if you want to change some configs in either of these two xml files then we need to restart NiFi to take those changes in to effect in NiFi instance.

Refer to this link regarding Process Group variables in NiFi and refer to this link regarding copying xml files into nifi lib.