Support Questions

Find answers, ask questions, and share your expertise

Does PutParquet processor support writing to Local Folder apart from writing to HDFS?

Explorer

I have a special case which is required to write out Parquet file into local directory.

I tried not to enter 'Hadoop Configuration Resources' but it will instantly throw exceptions.

If I entered "Hadoop Configuration Resources", then the output path is actually points to path inside the HDFS cluster.

So, I would like to know, Does PutParquet processor support writing to Local Folder?

1 ACCEPTED SOLUTION

Super Guru

I think you can provide a core-site.xml that refers to a local filesystem as the default FS rather than an HDFS file system by setting the fs.default.name property to file:/// (see the answer on this SO post for more details).

View solution in original post

4 REPLIES 4

Super Guru

I think you can provide a core-site.xml that refers to a local filesystem as the default FS rather than an HDFS file system by setting the fs.default.name property to file:/// (see the answer on this SO post for more details).

Matt is correct, and just to elaborate further... the Parquet API only allows writing through the Hadoop FileSystem, so the only way NiFi can write Parquet to local filesystem is through the configuration Matt mentioned above.

Explorer

Thanks all. One more question: How does PutParquet Processor connect to HDFS? By WebHDFS , HttpFS or Native ?

Native through the standard Hadoop Java client.

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.