question Re: Standard way of giving Hadoop Configuration Resources in PutHDFS in Archives of Support Questions (Read Only)

Standard way of giving Hadoop Configuration Resources in PutHDFS

dhieru — Tue, 10 Oct 2017 05:22:13 GMT

Hi All.

Thanks for the help

Working on PutHDFS, the NiFi is running on HDF cluster. I read that we need give the location of hdfs-site.xml and core-site.xml. Do I need to manually copy these xml config files over to HDF cluster and give the location?

or is there any other way? We have HDP cluster running and HDF cluster running. If i have to manually copy these files over to HDF cluster, is there a standard practice for it?

question1.png

Thanks All

Re: Standard way of giving Hadoop Configuration Resources in PutHDFS

Shu_ashu — Sun, 18 Aug 2019 03:06:20 GMT

Hi @dhieru singh, you can copy hdfs-site.xml,core-site.xml to nifi lib path and restart nifi, then you don't have to specify the path because nifi will load all the .xml from lib path.

path:-

/usr/hdf/current/nifi/lib

If you are having xml's in nifi lib path then you dont have to give any path for Hadoop Configuration Resources property.

(or)

you can use hdfs-site.cml,core-site.xml from

/etc/hdp/hdfs-site.xml,/etc/hdp/core-site.xml

and specify the path in the HDFS processor configurations then nifi will refer to those configuration resources from your specified path.

If you don't have xml's in nifi lib path then you have to give any path for Hadoop Configuration Resources property.

Re: Standard way of giving Hadoop Configuration Resources in PutHDFS

dhieru — Tue, 10 Oct 2017 19:21:15 GMT

@Shu Thanks a lot, this helps

Re: Standard way of giving Hadoop Configuration Resources in PutHDFS

alarsen — Fri, 17 Nov 2017 22:11:59 GMT

for me /usr/hdf/current/nifi/lib did not work.

HDF Version 2.1.1

Apache NiFi - Version 1.1.0.2.1.2.0-10

Re: Standard way of giving Hadoop Configuration Resources in PutHDFS

sshringi003 — Fri, 24 Aug 2018 20:12:33 GMT

Hi @Shu

I am running NiFi in localhost. I want to store data on server instead of local. So how to define the server property in

Hadoop Configuration Resources.

Bcz it requires ip address and path of the core-site.xml.

If you help it would be great.

Thanks,

Surendra

Re: Standard way of giving Hadoop Configuration Resources in PutHDFS

Shu_ashu — Sat, 25 Aug 2018 02:20:43 GMT

@Surendra Shringi

Copy the xml's in to your local nifi and then configure the processors to use those .xml files in the local.
If you are having kerberos enable env then you need to have the kerberos keytabs also in your local and change in nifi.properties to use those keytabs.

Re: Standard way of giving Hadoop Configuration Resources in PutHDFS

sshringi003 — Sat, 25 Aug 2018 12:38:53 GMT

Should i copy all the xml's file from server nifi conf to local nifi conf

server nifi/conf/.xml --> local nifi/conf/.xml

or which files need to be copy, bcz i did not get actual answer.

Pls ...

Re: Standard way of giving Hadoop Configuration Resources in PutHDFS

Shu_ashu — Mon, 27 Aug 2018 18:26:00 GMT

@Surendra Shringi

You don't have to copy the .xml files into nifi/conf directory instead you can keep them in some other directories then keep the path to the file in your processor configs.

These two files hdfs-site.xml,core-site.xml needs to be copied over to your local nifi.

Re: Standard way of giving Hadoop Configuration Resources in PutHDFS

sshringi003 — Wed, 29 Aug 2018 11:10:10 GMT

Thanks for your quick reply. I tried but unable to figure out the problem. The same error occurs while putting data into HBase.

Re: Standard way of giving Hadoop Configuration Resources in PutHDFS

asand3r — Thu, 19 Jun 2025 08:31:07 GMT

@Shu_ashu hi, could you answer one more question?.. What should I do, if we have many Hadoop clusters? Is it OK if I put hdfs-site.xml and core-site.xml for each cluster in lib directory with different names? E.g.:

./lib/hadoop1_hdfs-site.xml
./lib/hadoop1_core-site.xml
./lib/hadoop2_hdfs-site.xml
./lib/hadoop2_core-site.xml