Standard way of giving Hadoop Configuration Resources in PutHDFS

[#1]

Hi All.

Thanks for the help

Working on PutHDFS, the NiFi is running on HDF cluster. I read that we need give the location of hdfs-site.xml and core-site.xml. Do I need to manually copy these xml config files over to HDF cluster and give the location?

or is there any other way? We have HDP cluster running and HDF cluster running. If i have to manually copy these files over to HDF cluster, is there a standard practice for it?

question1.png

Thanks All

[#2 · Accepted solution]

Hi @dhieru singh, you can copy hdfs-site.xml,core-site.xml to nifi lib path and restart nifi, then you don't have to specify the path because nifi will load all the .xml from lib path.

path:-

/usr/hdf/current/nifi/lib

If you are having xml's in nifi lib path then you dont have to give any path for Hadoop Configuration Resources property.

(or)

you can use hdfs-site.cml,core-site.xml from

/etc/hdp/hdfs-site.xml,/etc/hdp/core-site.xml

and specify the path in the HDFS processor configurations then nifi will refer to those configuration resources from your specified path.

If you don't have xml's in nifi lib path then you have to give any path for Hadoop Configuration Resources property.

[#3]

@Shu Thanks a lot, this helps

[#4]

for me /usr/hdf/current/nifi/lib did not work.

HDF Version 2.1.1

Apache NiFi - Version 1.1.0.2.1.2.0-10

[#5]

Hi @Shu

I am running NiFi in localhost. I want to store data on server instead of local. So how to define the server property in

Hadoop Configuration Resources.

Bcz it requires ip address and path of the core-site.xml.

If you help it would be great.

Thanks,

Surendra

[#6]

@Surendra Shringi

Copy the xml's in to your local nifi and then configure the processors to use those .xml files in the local.
If you are having kerberos enable env then you need to have the kerberos keytabs also in your local and change in nifi.properties to use those keytabs.

[#7]

Should i copy all the xml's file from server nifi conf to local nifi conf

server nifi/conf/.xml --> local nifi/conf/.xml

or which files need to be copy, bcz i did not get actual answer.

Pls ...

[#8]

@Surendra Shringi

You don't have to copy the .xml files into nifi/conf directory instead you can keep them in some other directories then keep the path to the file in your processor configs.

These two files hdfs-site.xml,core-site.xml needs to be copied over to your local nifi.

[#9]

Thanks for your quick reply. I tried but unable to figure out the problem. The same error occurs while putting data into HBase.