Hive query exception from Java api

[#1]

I am trying to execute below query using java api

INSERT OVERWRITE LOCAL DIRECTORY '/home/bigframe/aps/temp' select * from teradata_demo_table

And I am getting below error

java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
        at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:296)
        at com.bigframe.perl.api.HiveReconTableCreate.doHiveTableCreation(HiveReconTableCreate.java:82)
        at com.bigframe.perl.api.HiveReconTableCreate.main(HiveReconTableCreate.java:142)

Same query is working from Hive CLI. Also "select * from teradata_demo_table" is working from java API.

Please suggest.

[#2]

Are all queries failing via java api or just this one? Can you post complete error stack? I see hive "doHiveTableCreation" method, did table get created successfully?

[#3]

@Pranay Vyas

Thanks for your quick response. Hive table created successfully by hive CLI . below is the table structure.

hive> desc teradata_demo_table;
OK
id                      int
col_date                string
col_time                string
col_timestamp           string
col_timestamp_tz        string
col_time_tz             string
col_interval_yr_month   string
col_interval_yr         string
col_interval_month      string
col_interval_day        string
col_interval_day_hr     string
col_period_date         string
col_period_timestamp    string
col_byteint             tinyint
col_smallint            smallint
col_integer             int
col_bigint              bigint
col_decimal             decimal(8,2)
col_float               string
col_char                char(7)
col_varchar             varchar(16384)
Time taken: 0.379 seconds, Fetched: 21 row(s)

Full Log:

2017-01-24 16:01:38,840 INFO  [HiveServer2-Background-Pool: Thread-10098]: exec.Utilities (Utilities.java:getBaseWork(400)) - PLAN PATH = hdfs://ctsc00691239901.cts.com:8020/tmp/hive/hive/ae98860b-bdbe-45a1-9212-172a3bae152b/hive_2017-01-24_16-01-38_673_8353564872225299216-27/-mr-10003/0cfdbbd9-00ba-4f02-a100-5287bdaedfb8/reduce.xml
2017-01-24 16:01:38,841 INFO  [HiveServer2-Background-Pool: Thread-10098]: exec.Utilities (Utilities.java:getBaseWork(466)) - File not found: File does not exist: /tmp/hive/hive/ae98860b-bdbe-45a1-9212-172a3bae152b/hive_2017-01-24_16-01-38_673_8353564872225299216-27/-mr-10003/0cfdbbd9-00ba-4f02-a100-5287bdaedfb8/reduce.xml
	at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71)
	at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61)
	...


2017-01-24 16:01:38,841 INFO  [HiveServer2-Background-Pool: Thread-10098]: exec.Utilities (Utilities.java:getBaseWork(467)) - No plan file found: hdfs://ctsc00691239901.cts.com:8020/tmp/hive/hive/ae98860b-bdbe-45a1-9212-172a3bae152b/hive_2017-01-24_16-01-38_673_8353564872225299216-27/-mr-10003/0cfdbbd9-00ba-4f02-a100-5287bdaedfb8/reduce.xml
2017-01-24 16:01:38,848 INFO  [HiveServer2-Background-Pool: Thread-10098]: hdfs.DFSClient (DFSClient.java:getDelegationToken(1047)) - Created HDFS_DELEGATION_TOKEN token 3924 for hive on 10.223.72.129:8020
2017-01-24 16:01:38,857 INFO  [HiveServer2-Background-Pool: Thread-10098]: mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(249)) - Cleaning up the staging area /user/hive/.staging/job_1484412561270_0059
2017-01-24 16:01:38,858 ERROR [HiveServer2-Background-Pool: Thread-10098]: exec.Task (SessionState.java:printError(962)) - Job Submission failed with exception 'java.io.IOException(java.lang.reflect.UndeclaredThrowableException)'
java.io.IOException: java.lang.reflect.UndeclaredThrowableException
	at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:892)
	at org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:86)
	at org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2291)
	...
Caused by: java.lang.reflect.UndeclaredThrowableException
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1727)
	at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:874)
	... 40 more
Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, status: 403, message: Forbidden
	at org.apache.hadoop.security.authentication.client.AuthenticatedURL.extractToken(AuthenticatedURL.java:274)
	at org.apache.hadoop.security.authentication.client.PseudoAuthenticator.authenticate(PseudoAuthenticator.java:77)
	at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:128)
	at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:214)
	...


2017-01-24 16:01:38,860 INFO  [ATS Logger 0]: hooks.ATSHook (ATSHook.java:createPostHookEvent(193)) - Received post-hook notification for :hive_20170124160138_1cd543d0-afe0-46ee-95c6-b9fc1d4440f6
2017-01-24 16:01:38,860 ERROR [HiveServer2-Background-Pool: Thread-10098]: ql.Driver (SessionState.java:printError(962)) - FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
...
2017-01-24 16:01:38,861 ERROR [HiveServer2-Background-Pool: Thread-10098]: operation.Operation (SQLOperation.java:run(209)) - Error running hive query: 
org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
	at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:315)
	at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:156)
	at org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:71)
	at org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:206)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
	at org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:218)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

All other commands are running fine. This is simple hive query where I want to store the query result in local directory. Please help.

[#4]

Is your cluster kerberized? Log says "Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, status: 403, message: Forbidden"

Can you check if you are running the job with user having valid kerberos ticket?

[#5]

@Pranay Vyas

This is kerberized cluster. And I am using hive principal and keytab for authentication. All create table and insert data query are working from java api. But for any select query I am getting this authentication error (Authentication failed, status: 403, message: Forbidden). Please help.

[#6]

@Pranay Vyas

I have analyzed the log and found that select query from any table getting authentication error (Authentication failed, status: 403, message: Forbidden). Apart from select query all other queries are working. I am not sure why I am getting authentication error for select query. Select query is able to get the result set but getting this issue while iterating through result set. Please help.

[#7]

@Arkaprova Saha

If you are running Java code via JDBC connection to Hive, there is no "LOCAL" directory (except of local mode, which I believe is not the case). Well, actually, there is local dir, but it is on a machine that accepts your query (HS2). Most probably you just don't have writing permissions for the user you are connecting with.

I would suggest:

1. run the same query just without "LOCAL". it will create files on HDFS.

2. pull files from HDFS (hdfs get, or hdfs getmerge, or Java API File system).

Let me know if that works for you.

[#8]

@Ed Berezitsky

Thanks for your reply. I have analyzed the log and found that select query from any table getting authentication error (Authentication failed, status: 403, message: Forbidden). Apart from select query all other queries are working. I am not sure why I am getting authentication error for select query. Select query is able to get the result set but getting this issue while iterating through result set. Please help.

[#9]

So, DDLs and DMLs are running successfully, but SELECT fails.

Check if you have write permissions on /tmp directory for user you are running query with (looks like it is "hive" user).

The difference between running CLI and HS2 APIs - your code is running on different nodes (edge vs master). so make sure you have the same setup for user "hive" on both of them.

[#10 · Accepted solution]

This issue has been fixed after setting below properties in kms-site.xml.

hadoop.kms.proxyuser.hive.users=*

hadoop.kms.proxyuser.hive.hosts =*

Now I am not getting any authentication error.