Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

GenericUDF not returning data

Highlighted

GenericUDF not returning data

New Contributor

Hello,

I'm trying to get a GenericUDF running, but it is not returning data. Info below.

Output:

SLF4J: Class path contains multiple SLF4J bindings. 
SLF4J: Found binding in [jar:file:/usr/hdp/2.3.0.0-2557/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] 
SLF4J: Found binding in [jar:file:/usr/hdp/2.3.0.0-2557/spark/lib/spark-assembly-1.3.1.2.3.0.0-2557-hadoop2.7.1.2.3.0.0-2557.jar!/org/slf4j/impl/StaticLoggerBinder.clas
s] 
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
WARNING: Use "yarn jar" to launch YARN applications. 
SLF4J: Class path contains multiple SLF4J bindings. 
SLF4J: Found binding in [jar:file:/usr/hdp/2.3.0.0-2557/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] 
SLF4J: Found binding in [jar:file:/usr/hdp/2.3.0.0-2557/spark/lib/spark-assembly-1.3.1.2.3.0.0-2557-hadoop2.7.1.2.3.0.0-2557.jar!/org/slf4j/impl/StaticLoggerBinder.class] 
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 
Logging initialized using configuration in file:/etc/hive/2.3.0.0-2557/0/hive-log4j.properties 
OK 
Time taken: 1.566 seconds 
Added [/usr/hdp/2.3.0.0-2557/hive/lib/hive-contrib.jar] to class path 
Added resources: [/usr/hdp/2.3.0.0-2557/hive/lib/hive-contrib.jar]
converting to local hdfs:///group1/dev/TestUDF.jar 
Added [/tmp/8c45e69d-cccd-4f27-ab88-b87a1d492c98_resources/TestUDF.jar] to class path 
Added resources: [hdfs:///group1/dev/TestUDF.jar] converting to local hdfs:///group1/dev/TestUDF.jar 
Added [/tmp/8c45e69d-cccd-4f27-ab88-b87a1d492c98_resources/TestUDF.jar] to class path 
Added resources: [hdfs:///group1/dev/TestUDF.jar] 
OK 
Time taken: 0.181 seconds
converting to local hdfs:///group1/dev/TestUDF.jar 
Added [/tmp/8c45e69d-cccd-4f27-ab88-b87a1d492c98_resources/TestUDF.jar] to class path 
Added resources: [hdfs:///group1/dev/TestUDF.jar] 
OK 
Time taken: 0.125 seconds 
OK 
= = = = = = = = = = = = = = = = 
(eval) Have 3 arguments.
arg = 346184729713596 
arg = 8939945817 
arg = 29684024 
= = = = = = = = = = = = = = = = 
Failed with exception java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Hive internal error. 
Time taken: 0.772 seconds 
WARN: The method class org.apache.commons.logging.impl.SLF4JLogFactory#release() was invoked. 
WARN: Please see http://www.slf4j.org/codes.html#release for an explanation. 
WARN: The method class org.apache.commons.logging.impl.SLF4JLogFactory#release() was invoked. 
WARN: Please see http://www.slf4j.org/codes.html#release for an explanation. 
WARN: The method class org.apache.commons.logging.impl.SLF4JLogFactory#release() was invoked. 
WARN: Please see http://www.slf4j.org/codes.html#release for an explanation. 
WARN: The method class org.apache.commons.logging.impl.SLF4JLogFactory#release() was invoked. 
WARN: Please see http://www.slf4j.org/codes.html#release for an explanation.

Java GenericUDF source:

public class DataScrubUDF extends GenericUDF {
        ListObjectInspector listInputObjectInspector;
        StringObjectInspector elementOI;
    private ArrayList ret;
    private GenericUDFUtils.ReturnObjectInspectorResolver returnOIResolver;
        public ObjectInspector initialize(ObjectInspector[] args) throws UDFArgumentException {
                // First check - do we have the right number of arguments?
                if (args.length < 1) {
                        throw new UDFArgumentLengthException(
                                        "Need at least 1 argument.");
                }
                ret = new ArrayList();

                for (int ix = 0; ix < args.length; ix++) {
                        ObjectInspector a = args[ix];
                        if (!(a instanceof StringObjectInspector)) {
                                throw new UDFArgumentException(" argument must be a string");
                        }
                        this.elementOI = (StringObjectInspector) a;
                       if (args[ix].getCategory() != ObjectInspector.Category.PRIMITIVE)
                                throw new UDFArgumentTypeException(0, "Only primitive type arguments are accepted");
                }
                returnOIResolver = new GenericUDFUtils.ReturnObjectInspectorResolver(
                                true);
                return returnOIResolver.get();
        }

        public Object evaluate(DeferredObject[] args) throws HiveException {
                System.err.println("= = = = = = = = = = = = = = = =");
        // Empty the return array (re-used between calls)
        ret.clear();
                System.err.println("(eval) Have " + args.length + " arguments.");
                if (args.length < 1)
                        return null;
                // If passed a null, return a null
                if (args[0].get() == null)
                        return null;
        Object[] et;
        et = new Object[args.length];
                for (int ix = 0; ix < args.length; ix++) {
               if (args.length < 1)
                        return null;
                // If passed a null, return a null
                if (args[0].get() == null)
                        return null;
        Object[] et;
        et = new Object[args.length];
                for (int ix = 0; ix < args.length; ix++) {
                        String arg = elementOI.getPrimitiveJavaObject(args[ix].get() );
                        System.err.println("arg = " + arg);
                et[ix] = (Object) arg;
                ret.add(arg);
                }
                System.err.println("= = = = = = = = = = = = = = = =");
                return ret;
        }

        @Override
        public String getDisplayString(String[] args) {
                String retString = "(getDisplay) ";
                for (int ix = 0; ix < args.length; ix++) {
                        retString = retString + " | " + args[ix]; 
                }
                return retString;
        }         
}

Please let me know if you need any more information.

Thanks,

Harlan...

1 REPLY 1
Highlighted

Re: GenericUDF not returning data

Mentor

@Harlan Bloom great question, please review some documentation on how to unit test UDFs in your IDE. Here's an example I followed. http://blog.matthewrathbone.com/2013/08/10/guide-to-writing-hive-udfs.html

it is recommended you test your UDFs with unit testing rather than in Hive, it will save you a lot of time.

Don't have an account?
Coming from Hortonworks? Activate your account here