Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Standard way of giving Hadoop Configuration Resources in PutHDFS

Solved Go to solution

Standard way of giving Hadoop Configuration Resources in PutHDFS

Contributor

Hi All.

Thanks for the help

Working on PutHDFS, the NiFi is running on HDF cluster. I read that we need give the location of hdfs-site.xml and core-site.xml. Do I need to manually copy these xml config files over to HDF cluster and give the location?

or is there any other way? We have HDP cluster running and HDF cluster running. If i have to manually copy these files over to HDF cluster, is there a standard practice for it?

question1.png

Thanks All

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Standard way of giving Hadoop Configuration Resources in PutHDFS

Super Guru

Hi @dhieru singh, you can copy hdfs-site.xml,core-site.xml to nifi lib path and restart nifi, then you don't have to specify the path because nifi will load all the .xml from lib path.

path:-

/usr/hdf/current/nifi/lib

If you are having xml's in nifi lib path then you dont have to give any path for Hadoop Configuration Resources property.

40762-screenshot.png

(or)

you can use hdfs-site.cml,core-site.xml from

/etc/hdp/hdfs-site.xml,/etc/hdp/core-site.xml

and specify the path in the HDFS processor configurations then nifi will refer to those configuration resources from your specified path.

If you don't have xml's in nifi lib path then you have to give any path for Hadoop Configuration Resources property.

40761-screenshot.png

8 REPLIES 8

Re: Standard way of giving Hadoop Configuration Resources in PutHDFS

Super Guru

Hi @dhieru singh, you can copy hdfs-site.xml,core-site.xml to nifi lib path and restart nifi, then you don't have to specify the path because nifi will load all the .xml from lib path.

path:-

/usr/hdf/current/nifi/lib

If you are having xml's in nifi lib path then you dont have to give any path for Hadoop Configuration Resources property.

40762-screenshot.png

(or)

you can use hdfs-site.cml,core-site.xml from

/etc/hdp/hdfs-site.xml,/etc/hdp/core-site.xml

and specify the path in the HDFS processor configurations then nifi will refer to those configuration resources from your specified path.

If you don't have xml's in nifi lib path then you have to give any path for Hadoop Configuration Resources property.

40761-screenshot.png

Re: Standard way of giving Hadoop Configuration Resources in PutHDFS

Contributor

@Shu Thanks a lot, this helps

Re: Standard way of giving Hadoop Configuration Resources in PutHDFS

Contributor

for me /usr/hdf/current/nifi/lib did not work.

HDF Version 2.1.1

Apache NiFi - Version 1.1.0.2.1.2.0-10

Re: Standard way of giving Hadoop Configuration Resources in PutHDFS

New Contributor

Hi @Shu

I am running NiFi in localhost. I want to store data on server instead of local. So how to define the server property in

Hadoop Configuration Resources.

Bcz it requires ip address and path of the core-site.xml.

If you help it would be great.

Thanks,

Surendra

Re: Standard way of giving Hadoop Configuration Resources in PutHDFS

Super Guru
@Surendra Shringi

Copy the xml's in to your local nifi and then configure the processors to use those .xml files in the local.
If you are having kerberos enable env then you need to have the kerberos keytabs also in your local and change in nifi.properties to use those keytabs.

Highlighted

Re: Standard way of giving Hadoop Configuration Resources in PutHDFS

New Contributor

Should i copy all the xml's file from server nifi conf to local nifi conf

server nifi/conf/.xml --> local nifi/conf/.xml

or which files need to be copy, bcz i did not get actual answer.

Pls ...

Re: Standard way of giving Hadoop Configuration Resources in PutHDFS

Super Guru
@Surendra Shringi

You don't have to copy the .xml files into nifi/conf directory instead you can keep them in some other directories then keep the path to the file in your processor configs.

These two files hdfs-site.xml,core-site.xml needs to be copied over to your local nifi.

Re: Standard way of giving Hadoop Configuration Resources in PutHDFS

New Contributor

Thanks for your quick reply. I tried but unable to figure out the problem. The same error occurs while putting data into HBase.