question Write in HDFS from Impala UDF in Support Questions

Write in HDFS from Impala UDF

masfworld — Fri, 16 Sep 2022 09:12:05 GMT

Is it possible write a buffer into HDFS from UDF Impala (C++ or Java)? So I'd like simulate UDTF in Impala. First call to UDF to write output in HDFS. Later, another query reads these data from HDFS.

If it is possible, it will be with libhdfs. How can I install libhdfs-dev in debian wheezy?

Thanks!!!

Regards.

Re: Write in HDFS from Impala UDF

masfworld — Fri, 07 Nov 2014 09:06:27 GMT

Hi.

I answer myself. With libhdfs is possible write to HDFS from UDF. I've tested it and it works fine!!!

Re: Write in HDFS from Impala UDF

NareshKollu — Wed, 02 Jan 2019 13:22:49 GMT

I have a similar scenario. Could you please let me know what kind of permissions are needed for Impala UDF to be able to write into HDFS and to which user (is it the user who is executing the UDF or technical user "impala"?)

Re: Write in HDFS from Impala UDF

Tim Armstrong — Wed, 02 Jan 2019 18:57:41 GMT

UDFs can do a lot of things because they run with the same privileges as the Impala process. However, doing things other than the usual computations in the UDF, like accessing filesystems or external services, can compromise the performance and stability of your system. So you do this at your own risk. In the future we may lock down UDFs more and prevent them from doing things like accessing HDFS.