Impala UDF unable to load the .so file from HDFS

parthk — Fri, 09 Oct 2020 02:47:15 GMT

Hello Team,

We are planning to Impala UDFs and UDAs. To try things out we started by exploring the examples given in the cloudera github repo.

After building the .so file using the make utility when I try to create the function using the following

create function has_vowels (string) returns boolean location '/user/hive/udfs/libudfsample.so' symbol='HasVowels';

We are getting the following error

ERROR: AnalysisException: Could not load binary: /<hdfs_path>/udfs/libudfsample.so Unable to load /var/lib/impala/udfs/libudfsample.9909.1.so dlerror: /opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/impala/lib/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /var/lib/impala/udfs/libudfsample.9909.1.so)

Following are the error logs

org.apache.impala.common.AnalysisException: Could not load binary: /user/parth.khatwani/udfs/libudfsample.so Unable to load /var/lib/impala/udfs/libudfsample.9909.1.so dlerror: /opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/impala/lib/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /var/lib/impala/udfs/libudfsample.9909.1.so) at org.apache.impala.catalog.Function.lookupSymbol(Function.java:442) at org.apache.impala.analysis.CreateUdfStmt.analyze(CreateUdfStmt.java:92) at org.apache.impala.analysis.AnalysisContext.analyze(AnalysisContext.java:451) at org.apache.impala.analysis.AnalysisContext.analyzeAndAuthorize(AnalysisContext.java:421) at org.apache.impala.service.Frontend.doCreateExecRequest(Frontend.java:1285) at org.apache.impala.service.Frontend.getTExecRequest(Frontend.java:1252) at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1222) at org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:167) I1009 02:42:52.638576 15136 status.cc:124] 3c48433373076c77:30eff6ff00000000] AnalysisException: Could not load binary: /user/parth.khatwani/udfs/libudfsample.so Unable to load /var/lib/impala/udfs/libudfsample.9909.1.so

I am unable to figure out what's wrong here.

Regards

Parth

Re: Impala UDF unable to load the .so file from HDFS

Tim Armstrong — Fri, 09 Oct 2020 17:28:45 GMT

What OS and compiler version are you using to build the UDF? This looks like it is probably a consequence of it being built with a newer gcc version than the one use to build Impala (gcc 4.9.2)

Re: Impala UDF unable to load the .so file from HDFS

parthk — Mon, 12 Oct 2020 06:53:22 GMT

@Tim Armstrong I am using gcc version 5.4.0 and OS is ubuntu 16.04 xenial. Will it work If I compile it with 4.9.2?

Re: Impala UDF unable to load the .so file from HDFS

Tim Armstrong — Tue, 13 Oct 2020 22:44:23 GMT

@parthkyeah I'd expect so. Sometimes this C++ inter-version compatibility is a bear.

Re: Impala UDF unable to load the .so file from HDFS

parthk — Thu, 15 Oct 2020 11:13:28 GMT

@Tim Armstrong it worked like charm after changing the gcc version. Thanks

question Re: Impala UDF unable to load the .so file from HDFS in Support Questions

Impala UDF unable to load the .so file from HDFS

Re: Impala UDF unable to load the .so file from HDFS

Re: Impala UDF unable to load the .so file from HDFS

Re: Impala UDF unable to load the .so file from HDFS

Re: Impala UDF unable to load the .so file from HDFS