Write in HDFS from Impala UDF

[#1]

Hi

Is it possible write a buffer into HDFS from UDF Impala (C++ or Java)? So I'd like simulate UDTF in Impala. First call to UDF to write output in HDFS. Later, another query reads these data from HDFS.

If it is possible, it will be with libhdfs. How can I install libhdfs-dev in debian wheezy?

Thanks!!!

Regards.

[#2 · Accepted solution]

Hi.

I answer myself. With libhdfs is possible write to HDFS from UDF. I've tested it and it works fine!!!

[#3]

I have a similar scenario. Could you please let me know what kind of permissions are needed for Impala UDF to be able to write into HDFS and to which user (is it the user who is executing the UDF or technical user "impala"?)

[#4]

UDFs can do a lot of things because they run with the same privileges as the Impala process. However, doing things other than the usual computations in the UDF, like accessing filesystems or external services, can compromise the performance and stability of your system. So you do this at your own risk. In the future we may lock down UDFs more and prevent them from doing things like accessing HDFS.