question Re: Does PutParquet processor support writing to Local Folder apart from writing to HDFS? in Archives of Support Questions (Read Only)

Does PutParquet processor support writing to Local Folder apart from writing to HDFS?

ctgj0004 — Fri, 09 Mar 2018 14:49:21 GMT

I have a special case which is required to write out Parquet file into local directory.

I tried not to enter 'Hadoop Configuration Resources' but it will instantly throw exceptions.

If I entered "Hadoop Configuration Resources", then the output path is actually points to path inside the HDFS cluster.

So, I would like to know, Does PutParquet processor support writing to Local Folder?

Re: Does PutParquet processor support writing to Local Folder apart from writing to HDFS?

mburgess — Fri, 09 Mar 2018 22:34:08 GMT

I think you can provide a core-site.xml that refers to a local filesystem as the default FS rather than an HDFS file system by setting the fs.default.name property to file:/// (see the answer on this SO post for more details).

Re: Does PutParquet processor support writing to Local Folder apart from writing to HDFS?

bbende — Fri, 09 Mar 2018 23:29:13 GMT

Matt is correct, and just to elaborate further... the Parquet API only allows writing through the Hadoop FileSystem, so the only way NiFi can write Parquet to local filesystem is through the configuration Matt mentioned above.

Re: Does PutParquet processor support writing to Local Folder apart from writing to HDFS?

ctgj0004 — Mon, 12 Mar 2018 14:03:25 GMT

Thanks all. One more question: How does PutParquet Processor connect to HDFS? By WebHDFS , HttpFS or Native ?

Re: Does PutParquet processor support writing to Local Folder apart from writing to HDFS?

bbende — Mon, 12 Mar 2018 20:04:13 GMT

Native through the standard Hadoop Java client.