Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

UDF missing after Impala restart

UDF missing after Impala restart

Expert Contributor

I'm installing a Hive UDF (in Java) on Impala with the HDFS location of the JAR but whenever Impala is restarted the UDFs are missing. I'm using CDH 5.3.2 with Impala 2.1.2. 

 

Please let me know if there is anyway to not loose the UDF or trigger the install after every restart.

 

Thanks!

22 REPLIES 22

Re: UDF missing after Impala restart

Master Collaborator

I'm afraid the UDFs are lost after restarting the catalogd. One way to avoid the issue is to only selectively restart the impalad and/or statestored if that is suffucient in your setup.

 

Btw, we use https://issues.cloudera.org/browse/IMPALA-1748 to track the feature of persistent UDFs.

Re: UDF missing after Impala restart

New Contributor

I'm also running into this issue with Java UDFs stored in HDFS. The current Cloudera docs however state that this should work:

Currently, Impala UDFs and UDAs written in C++ are not persisted in the metastore database. Information about these functions is held in the memory of thecatalogd daemon. You must reload them by running the CREATE FUNCTION statements again each time you restart the catalogd daemon. This limitation does not apply to Impala UDFs and UDAs written in Java.

 Are the docs incorrect in this case or am I doing something stupid?

Re: UDF missing after Impala restart

Master Collaborator

The docs are incorrect. I've already asked for them to be fixed. Thanks for pointing out this issue!

Re: UDF missing after Impala restart

Explorer

Hi,

 

Any updates on the above issue.

I have created hive udf and using with impala to query hbase.

i am also facing the same issue. Udfs are getting dropped after restarting daemons.

We have around 50 udfs which needs to be created after scheduled every cluster restart or impala service restart.

 

I cant see any progress in https://issues.cloudera.org/browse/IMPALA-1748.

 

Is there any temporary solution to avoid create udfs again and again after restarting ?

 

Thanks !!!

Re: UDF missing after Impala restart

Contributor

We have no updates to IMPALA-1748 to report at this time.

Re: UDF missing after Impala restart

Contributor

Hi Guys,

 

 

I am also facing similar issue, as we restart impala service the UDF's are gone and we have to recreate them manually or through a script, Is there any fix or a way to avoid manually creating them after restart as it is becoming an important and complicated issue for us. Please help me out with this. Also is there any way to add them permanently.

Re: UDF missing after Impala restart

Master Collaborator

We're tracking the issue with https://issues.cloudera.org/browse/IMPALA-1748. We're aware that a number of people are hitting this issue and take that into account when prioritizing it relative to other bugfixes and features.

Re: UDF missing after Impala restart

Contributor

Today I installed CDH quickstart vm 5.5 which has Impala 2.3.0+cdh5.5.0+0, But still I am seeing same issue, after restart of Impala service the UDF's are gone and again I have to manually create it.

 

Can you please tell me when will be this fixed and what will that CDH and Impala release be. Its very urgent issue, as our code is sometimes failing in production.

Re: UDF missing after Impala restart

Cloudera Employee

There is a patch for this under review now. There is a release scheduled for sometime around April-May next year and the patch should be included.