Hive query failed over JDBC.

[#1]

I am working on hive JDBC client.I am able to run normal queries like select * from table, show database etc. But when I am trying to run queries like insert into ,select count(*) it simply fails with following exception.

java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:282) at org.apache.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:378) at TablePerformance.countItems(TablePerformance.java:51) at TablePerformance.main(TablePerformance.java:63)

I have checked hive server logs and it also shows same error message as above. Is there any way I can set hive.execution.engine over jdbc connectivity ?

Any kind of help would be greatly appreciated.

[#2]

Hi @Girish Chaudhari

It's hard to tell what the root cause is based on the above error.

Can you provide yarn application log? You can get it by running 'yarn logs -applicationId [your application_id]'

[#3]

@Takahiko Saito Stack trace just after jdbc query failure.

Container exited with a non-zero exit code 127
Failing this attempt. Failing the application.
2016-07-28 00:11:22,769 INFO  rmapp.RMAppImpl (RMAppImpl.java:handle(767)) - application_1466711437570_0051 State change from FINAL_SAVING to FAILED
2016-07-28 00:11:22,769 WARN  resourcemanager.RMAuditLogger (RMAuditLogger.java:logFailure(323)) - USER="girish"        OPERATION=Application Finished - Failed TARGET=RMAppManager     RESULT=FAILURE  DESCRIPTION=App failed with state: FAILED    PERMISSIONS=Application application_1466711437570_0051 failed 2 times due to AM Container for appattempt_1466711437570_0051_000002 exited with  exitCode: 127
For more detailed output, check application tracking page then click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_e02_1466711437570_0051_02_000001
Exit code: 127
Exception message: /bin/bash: /mnt/resource/usercache/girish/appcache/application_1466711437570_0051/container_e02_1466711437570_0051_02_000001/default_container_executor_session.sh: No such file or directory


Stack trace: ExitCodeException exitCode=127: /bin/bash: /mnt/resource/usercache/girish/appcache/application_1466711437570_0051/container_e02_1466711437570_0051_02_000001/default_container_executor_session.sh: No such file or directory


        at org.apache.hadoop.util.Shell.runCommand(Shell.java:576)
        at org.apache.hadoop.util.Shell.run(Shell.java:487)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:753)
        at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:303)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)




Container exited with a non-zero exit code 127
Failing this attempt. Failing the application.  APPID=application_1466711437570_0051
2016-07-28 00:11:22,769 INFO  capacity.ParentQueue (ParentQueue.java:removeApplication(372)) - Application removed - appId: application_1466711437570_0051 user: "girish" leaf-queue of parent: root #applications: 0
2016-07-28 00:11:23,446 INFO  resourcemanager.RMAuditLogger (RMAuditLogger.java:logSuccess(170)) - USER="girish"        IP=10.0.0.5     OPERATION=Kill Application Request      TARGET=ClientRMService  RESULT=SUCCESS  APPID=application_1466711437570_0051 CALLERCONTEXT=HIVE_SSN_ID:d8a476e9-84c4-4633-a980-3f8a9dd7dac8

[#4]

Hi @Girish Chaudhari

Thanks for the stack trace. Now we know why it failed. It's because the container cannot be launched because default_container_executor_session.sh doesn't exist at /mnt/resource/usercache/girish/appcache/application_1466711437570_0051/container_e02_1466711437570_0051_02_000001

As to why this happened, that still needs to be investigated. Will update if I find something.

[#5]

@Girish Chaudhari How much memory do you have on your cluster (on each node)?

A simple select statement is simply able to stream results back to JDBC client. count() however, needs to work and requires memory and I am just wondering what is assigned here?

[#6]

@mqureshi I am able to run same query through hive view and hive cli but not through jdbc .

Cluster configuration is as below:

Name node and Secondary node (8 Core / 28 GB RAM each) 
3 Data Nodes (8 Core / 28 GB RAM each)

I tried to set hive.execution.engine=tez by appending it to jdbc connect url but it doesn't work.

[#7 · Accepted solution]

I solved it by passing the below connection objects with my user name.

con = DriverManager.getConnection("jdbc:hive2://host:10000/default?hive.execution.engine=tez", "<user-name>","");

Thanks you all of you for your help and advice.

[#8]

Dear @Girish Chaudhari,

Unfortunately, I have faced this issue as well, but I could not understand the BEST ANSWER submitted by you. If it is possible, please help me.

thanks a million,

Ali

[#9]

Hi Ali,

Did you tried to pass "hive.execution.engine=tez" as session config property in JDBC connection URL. Please try that and let me if that works.

e.g.

con = DriverManager.getConnection("jdbc:hive2://host:10000/default?hive.execution.engine=tez")

[#10]

I had also been facing that issue in hive, but I solved by setting "use.hive.interactive.mode" to true and restart the Hive View to run in llap mode.

thank you