Reply
Highlighted
Explorer
Posts: 13
Registered: ‎04-08-2014
Accepted Solution

Write in HDFS from Impala UDF

Hi

 

Is it possible write a buffer into HDFS from UDF Impala (C++ or Java)? So I'd like simulate UDTF in Impala. First call to UDF to write output in HDFS. Later, another query reads these data from HDFS.

 

If it is possible, it will be with libhdfs. How can I install libhdfs-dev in debian wheezy?

 

 

Thanks!!!

Regards.

 

Explorer
Posts: 13
Registered: ‎04-08-2014

Re: Write in HDFS from Impala UDF

Hi.

 

I answer myself. With libhdfs is possible write to HDFS from UDF. I've tested it and it works fine!!!

New Contributor
Posts: 1
Registered: ‎12-17-2018

Re: Write in HDFS from Impala UDF

I have a similar scenario. Could you please let me know what kind of permissions are needed for Impala UDF to be able to write into HDFS and to which user (is it the user who is executing the UDF or technical user "impala"?)

Cloudera Employee
Posts: 395
Registered: ‎07-29-2015

Re: Write in HDFS from Impala UDF

UDFs can do a lot of things because they run with the same privileges as the Impala process. However, doing things other than the usual computations in the UDF, like accessing filesystems or external services, can compromise the performance and stability of your system. So you do this at your own risk. In the future we may lock down UDFs more and prevent them from doing things like accessing HDFS.
Announcements