Support Questions
Find answers, ask questions, and share your expertise

Hive UDF creates issue for insert query

I am checking how to create a custom UDF in Hive. I have created a custom UDF which does nothing, just returns the given text as it is.

  • I am able to load this JAR in hive without any issue
  • I am also able to create function from this jar and I am able to execute this function.

Problem

if this Jar is added I can't execute load table/hdfs from another table. following simple query fails.

insert into demo1 select * from demo;

Stacktrace:

Vertex failed, vertexName=Map 1, vertexId=vertex_1523501275422_0010_3_00, diagnostics=[Task failed, taskId=task_1523501275422_0010_3_00_000000, diagnostics=[TaskAttempt 0 failed, info=[Container container_1523501275422_0010_01_000007 finished with diagnostics set to [Container completed. ]], TaskAttempt 1 failed, info=[Container container_1523501275422_0010_01_000008 finished with diagnostics set to [Container completed. ]], TaskAttempt 2 failed, info=[Container container_1523501275422_0010_01_000009 finished with diagnostics set to [Container completed. ]], TaskAttempt 3 failed, info=[Container container_1523501275422_0010_01_000010 finished with diagnostics set to [Container completed. ]]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0, Vertex vertex_1523501275422_0010_3_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE]
DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0
<br>

Please note that both the table has same structure.

If I remove the jar then I can execute above query without any issue.

Code: ca.abc.demo

public class demo extends UDF {
    public String evaluate(String s) {
        if (s == null) {
            return null;
        }else{
            return s;
        }
    }
}<br>

pom.xml

<?xml version="1.0" encoding="UTF-8"?><project xmlns="http://maven.apache.org/POM/4.0.0"xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">    <modelVersion>4.0.0</modelVersion>    <groupId>ca.cantire</groupId>    <artifactId>hive-normalize</artifactId>    <version>1.0</version>    <dependencies><!-- https://mvnrepository.com/artifact/org.apache.hive/hive-exec --><dependency>            <groupId>org.apache.hive</groupId>            <artifactId>hive-exec</artifactId>            <version>2.3.3</version>        </dependency>    </dependencies>    <build>        <pluginManagement>            <plugins>                <plugin>                    <groupId>org.apache.maven.plugins</groupId>                    <artifactId>maven-surefire-plugin</artifactId>                    <version>2.8</version>                </plugin>                <plugin>                    <artifactId>maven-assembly-plugin</artifactId>                    <configuration>                        <archive>                            <manifest>                                <mainClass>ca.cantire.demo</mainClass>                            </manifest>                        </archive>                        <descriptorRefs>                            <descriptorRef>jar-with-dependencies</descriptorRef>                        </descriptorRefs>                    </configuration>                </plugin>            </plugins>        </pluginManagement>    </build></project>
1 REPLY 1

Contributor

@Gaurang Shah

Please check for the exact error in the Yarn application log corresponding to query run. The Yarn app logs should give us a better understanding of Vertex failure issue. Attache them here, if you can.

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.