Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Does PutParquet processor support writing to Local Folder apart from writing to HDFS?

avatar
Explorer

I have a special case which is required to write out Parquet file into local directory.

I tried not to enter 'Hadoop Configuration Resources' but it will instantly throw exceptions.

If I entered "Hadoop Configuration Resources", then the output path is actually points to path inside the HDFS cluster.

So, I would like to know, Does PutParquet processor support writing to Local Folder?

1 ACCEPTED SOLUTION

avatar
Master Guru

I think you can provide a core-site.xml that refers to a local filesystem as the default FS rather than an HDFS file system by setting the fs.default.name property to file:/// (see the answer on this SO post for more details).

View solution in original post

4 REPLIES 4

avatar
Master Guru

I think you can provide a core-site.xml that refers to a local filesystem as the default FS rather than an HDFS file system by setting the fs.default.name property to file:/// (see the answer on this SO post for more details).

avatar
Master Guru

Matt is correct, and just to elaborate further... the Parquet API only allows writing through the Hadoop FileSystem, so the only way NiFi can write Parquet to local filesystem is through the configuration Matt mentioned above.

avatar
Explorer

Thanks all. One more question: How does PutParquet Processor connect to HDFS? By WebHDFS , HttpFS or Native ?

avatar
Master Guru

Native through the standard Hadoop Java client.