question Re: Nifi not using HDFS in Archives of Support Questions (Read Only)

Nifi not using HDFS

camatulli — Wed, 11 Jan 2017 22:42:19 GMT

I am trying to write a HDF process which takes a feed from the listenSMTP receiver, and puts (PutHDFS) files into a folder in HDFS. For some reason, the PutHDFS is not using HDFS and instead putting in a local folder to the HDF setup. I've confirmed the hdfs-site.xml file is referenced correctly. I've also tried the FetchHDFS and ListHDFS options, and they all seem to do similar activity. Anyone have ideas on this? why would a PutHDFS put locally instead of in HDFS

Re: Nifi not using HDFS

bbende — Thu, 12 Jan 2017 01:03:24 GMT

You need to provide core-site.xml and hdfs-site.xml, the core-site.xml should have a default filesystem provided like this:

<name>fs.defaultFS</name>

<value>hdfs://hostname</value>

</property>

Re: Nifi not using HDFS

camatulli — Thu, 12 Jan 2017 05:04:11 GMT

the nifi processors for hdfs only have a spot for 1 xml file, and it wants the hdfs-site.xml. where to I reference the core-site.xml?

Re: Nifi not using HDFS

bbende — Thu, 12 Jan 2017 05:24:25 GMT

It takes a comma separated list of files so you can specify:

/etc/hadoop/conf/core-site.xml,/etc/hadoop/conf/hdfs-site.xml

Obviously using the appropriate paths on your file system.

Re: Nifi not using HDFS

camatulli — Fri, 13 Jan 2017 00:40:23 GMT

Thank you! that was the solution! I fought with that for two weeks with no error message telling me it was missing anything.