Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Write in HDFS from Impala UDF

Solved Go to solution

Write in HDFS from Impala UDF

Explorer

Hi

 

Is it possible write a buffer into HDFS from UDF Impala (C++ or Java)? So I'd like simulate UDTF in Impala. First call to UDF to write output in HDFS. Later, another query reads these data from HDFS.

 

If it is possible, it will be with libhdfs. How can I install libhdfs-dev in debian wheezy?

 

 

Thanks!!!

Regards.

 

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Write in HDFS from Impala UDF

Explorer

Hi.

 

I answer myself. With libhdfs is possible write to HDFS from UDF. I've tested it and it works fine!!!

3 REPLIES 3

Re: Write in HDFS from Impala UDF

Explorer

Hi.

 

I answer myself. With libhdfs is possible write to HDFS from UDF. I've tested it and it works fine!!!

Highlighted

Re: Write in HDFS from Impala UDF

New Contributor

I have a similar scenario. Could you please let me know what kind of permissions are needed for Impala UDF to be able to write into HDFS and to which user (is it the user who is executing the UDF or technical user "impala"?)

Re: Write in HDFS from Impala UDF

Master Collaborator
UDFs can do a lot of things because they run with the same privileges as the Impala process. However, doing things other than the usual computations in the UDF, like accessing filesystems or external services, can compromise the performance and stability of your system. So you do this at your own risk. In the future we may lock down UDFs more and prevent them from doing things like accessing HDFS.
Don't have an account?
Coming from Hortonworks? Activate your account here