Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Standard way of giving Hadoop Configuration Resources in PutHDFS

avatar
Expert Contributor

Hi All.

Thanks for the help

Working on PutHDFS, the NiFi is running on HDF cluster. I read that we need give the location of hdfs-site.xml and core-site.xml. Do I need to manually copy these xml config files over to HDF cluster and give the location?

or is there any other way? We have HDP cluster running and HDF cluster running. If i have to manually copy these files over to HDF cluster, is there a standard practice for it?

question1.png

Thanks All

1 ACCEPTED SOLUTION

avatar
Master Guru

Hi @dhieru singh, you can copy hdfs-site.xml,core-site.xml to nifi lib path and restart nifi, then you don't have to specify the path because nifi will load all the .xml from lib path.

path:-

/usr/hdf/current/nifi/lib

If you are having xml's in nifi lib path then you dont have to give any path for Hadoop Configuration Resources property.

40762-screenshot.png

(or)

you can use hdfs-site.cml,core-site.xml from

/etc/hdp/hdfs-site.xml,/etc/hdp/core-site.xml

and specify the path in the HDFS processor configurations then nifi will refer to those configuration resources from your specified path.

If you don't have xml's in nifi lib path then you have to give any path for Hadoop Configuration Resources property.

40761-screenshot.png

View solution in original post

8 REPLIES 8

avatar
Master Guru

Hi @dhieru singh, you can copy hdfs-site.xml,core-site.xml to nifi lib path and restart nifi, then you don't have to specify the path because nifi will load all the .xml from lib path.

path:-

/usr/hdf/current/nifi/lib

If you are having xml's in nifi lib path then you dont have to give any path for Hadoop Configuration Resources property.

40762-screenshot.png

(or)

you can use hdfs-site.cml,core-site.xml from

/etc/hdp/hdfs-site.xml,/etc/hdp/core-site.xml

and specify the path in the HDFS processor configurations then nifi will refer to those configuration resources from your specified path.

If you don't have xml's in nifi lib path then you have to give any path for Hadoop Configuration Resources property.

40761-screenshot.png

avatar
Expert Contributor

@Shu Thanks a lot, this helps

avatar
Contributor

for me /usr/hdf/current/nifi/lib did not work.

HDF Version 2.1.1

Apache NiFi - Version 1.1.0.2.1.2.0-10

avatar
Rising Star

Hi @Shu

I am running NiFi in localhost. I want to store data on server instead of local. So how to define the server property in

Hadoop Configuration Resources.

Bcz it requires ip address and path of the core-site.xml.

If you help it would be great.

Thanks,

Surendra

avatar
Master Guru
@Surendra Shringi

Copy the xml's in to your local nifi and then configure the processors to use those .xml files in the local.
If you are having kerberos enable env then you need to have the kerberos keytabs also in your local and change in nifi.properties to use those keytabs.

avatar
Rising Star

Should i copy all the xml's file from server nifi conf to local nifi conf

server nifi/conf/.xml --> local nifi/conf/.xml

or which files need to be copy, bcz i did not get actual answer.

Pls ...

avatar
Master Guru
@Surendra Shringi

You don't have to copy the .xml files into nifi/conf directory instead you can keep them in some other directories then keep the path to the file in your processor configs.

These two files hdfs-site.xml,core-site.xml needs to be copied over to your local nifi.

avatar
Rising Star

Thanks for your quick reply. I tried but unable to figure out the problem. The same error occurs while putting data into HBase.