Support Questions
Find answers, ask questions, and share your expertise

Pig UDF with External Jars

Pig UDF with External Jars

Explorer

Dear Experts,

 

I have read that external jars or libraries used in Java Pig UDFs can be registered the following way:

* REGISTER ../path/library.jar

* pig -Dpig.additional.jars=/local/path/to/your.jar

 

However I was writing a my SHA1 UDF based on Apache Commons Codec and wasn't succesful in registering this. I had the error that the method sha1hex could not be found. I had the following maven dependencies:

 

<dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-core</artifactId>
            <version>0.20.2</version>
        </dependency>
        <dependency>
            <groupId>org.apache.pig</groupId>
            <artifactId>piggybank</artifactId>
            <version>0.12.1</version>
        </dependency>

<dependency>
    <groupId>commons-codec</groupId>
    <artifactId>commons-codec</artifactId>
    <version>1.9</version>
</dependency>

<dependency>
    <groupId>org.apache.commons</groupId>
    <artifactId>commons-lang3</artifactId>
    <version>3.3.2</version>
</dependency>

 

Maybe the issue lies in dependency conflicts, both Apache commons-lang3 and commons-codec are also contained in the piggybank dependency; but these are pretty old. Eventually I ended up deleting the two apache dependencies and use the libs/jars already contained in Piggybank.

 

Why wasn't I able to REGISTER my external jars? How can I register them succesfully?

 

thanks for your help!

 

Greetings,

Mark

1 REPLY 1
Highlighted

Re: Pig UDF with External Jars

Master Guru
Please post your Pig grunt log, which will carry the specific exception that prevented your JAR from being registered.