Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hive to HDFS export fails with custom UDF

Hive to HDFS export fails with custom UDF

New Contributor

Hi Guys,

I have written a custom UDF function which works fine if I run that in select query. However, if I try to export data to HDFS using select query as custom UDF is part of that query, it fails.

Following query gets executed successfully.

CREATE temporary FUNCTION normalize as 'ca.test.Normalize' USING JAR 'hdfs://hadoopdev/tmp/udf/normalizer-1.0-jar-with-dependencies.jar';
select normalize(test_desc, 'NFC') from temp.test_special_char;


Following query Fails

CREATE temporary FUNCTION normalize as 'ca.test.Normalize' USING JAR 'hdfs://hadoopdev/tmp/udf/normalizer-1.0-jar-with-dependencies.jar';
INSERT OVERWRITE  DIRECTORY '/tmp/test_special_char/'
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '|'
LINES TERMINATED BY '\n'
NULL DEFINED AS ''
select normalize(test_desc, 'NFC') from temp.test_special_char;

StackTrace

INFO  : Tez session hasn't been created yet. Opening session
ERROR : Failed to execute tez graph.
org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_1519224124029_102976 failed 2 times due to AM Container for appattempt_1519224124029_102976_000002 exited with  exitCode: 255
For more detailed output, check the application tracking page: http://abc:8088/cluster/app/application_1519224124029_102976 Then click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_e71_1519224124029_102976_02_000001
Exit code: 255
Stack trace: org.apache.hadoop.yarn.server.nodemanager.containermanager.runtime.ContainerExecutionException: Launch container failed
at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DefaultLinuxContainerRuntime.launchContainer(DefaultLinuxContainerRuntime.java:109)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime.launchContainer(DelegatingLinuxContainerRuntime.java:89)
at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:392)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:317)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:83)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Shell output: main : command provided 1
main : run as user is gaurang.shah
main : requested yarn user is gaurang.shah
Getting exit code file...
Creating script paths...
Writing pid file...
Writing to tmp file /data/10/hadoop/yarn/local/nmPrivate/application_1519224124029_102976/container_e71_1519224124029_102976_02_000001/container_e71_1519224124029_102976_02_000001.pid.tmp
Writing to cgroup task files...
Creating local dirs...
Launching container...
Getting exit code file...
Creating script paths...
Container exited with a non-zero exit code 255
Failing this attempt. Failing the application.
at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:779)
at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:217)
at org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:272)
at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:152)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1745)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1491)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1289)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1156)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1151)
at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:197)
at org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:76)
at org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:253)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:264)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask (state=08S01,code=1)
1 REPLY 1
Highlighted

Re: Hive to HDFS export fails with custom UDF

Contributor
@Gaurang Shah

Can you please verify if you have proper permissions on '/tmp/test_special_char/' directory.