Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Write in HDFS from Impala UDF

avatar
Contributor

Hi

 

Is it possible write a buffer into HDFS from UDF Impala (C++ or Java)? So I'd like simulate UDTF in Impala. First call to UDF to write output in HDFS. Later, another query reads these data from HDFS.

 

If it is possible, it will be with libhdfs. How can I install libhdfs-dev in debian wheezy?

 

 

Thanks!!!

Regards.

 

1 ACCEPTED SOLUTION

avatar
Contributor

Hi.

 

I answer myself. With libhdfs is possible write to HDFS from UDF. I've tested it and it works fine!!!

View solution in original post

3 REPLIES 3

avatar
Contributor

Hi.

 

I answer myself. With libhdfs is possible write to HDFS from UDF. I've tested it and it works fine!!!

avatar
New Contributor

I have a similar scenario. Could you please let me know what kind of permissions are needed for Impala UDF to be able to write into HDFS and to which user (is it the user who is executing the UDF or technical user "impala"?)

avatar
UDFs can do a lot of things because they run with the same privileges as the Impala process. However, doing things other than the usual computations in the UDF, like accessing filesystems or external services, can compromise the performance and stability of your system. So you do this at your own risk. In the future we may lock down UDFs more and prevent them from doing things like accessing HDFS.