Member since
05-19-2016
93
Posts
17
Kudos Received
2
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 5286 | 01-30-2017 07:34 AM | |
| 3645 | 09-14-2016 10:31 AM |
01-30-2017
06:40 AM
@rguruvannagari Thanks a lot. After setting hive property , this issue is resolved.
... View more
01-30-2017
06:15 AM
I am trying to execute below hive select query using java api in kerberised cluster using hive principal and keytab. Apart from select query all other queries (create, drop,load) running fine. select * from abc90 And getting below authentication error . org.apache.hive.service.cli.HiveSQLException: java.io.IOException: java.io.IOException: java.lang.reflect.UndeclaredThrowableException
at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:352)
at org.apache.hive.service.cli.operation.OperationManager.getOperationNextRowSet(OperationManager.java:223)
at org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:716)
at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
at com.sun.proxy.$Proxy20.fetchResults(Unknown Source)
at org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:456)
at org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:672)
at org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1557)
at org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1542)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:562)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: java.io.IOException: java.lang.reflect.UndeclaredThrowableException
at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:512)
at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:419)
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:143)
at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1745)
at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:347)
... 24 more
Caused by: java.io.IOException: java.lang.reflect.UndeclaredThrowableException
at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:892)
at org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:86)
at org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2291)
at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:121)
at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:100)
at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:80)
at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:206)
at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:315)
at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextSplits(FetchOperator.java:367)
at org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:299)
at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:450)
... 28 more
Caused by: java.lang.reflect.UndeclaredThrowableException
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1727)
at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:874)
... 38 more
Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, status: 403, message: Forbidden
at org.apache.hadoop.security.authentication.client.AuthenticatedURL.extractToken(AuthenticatedURL.java:274)
at org.apache.hadoop.security.authentication.client.PseudoAuthenticator.authenticate(PseudoAuthenticator.java:77)
at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:128)
at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:214)
at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:128)
at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:215)
at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:285)
at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.getDelegationToken(DelegationTokenAuthenticator.java:166)
at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.getDelegationToken(DelegationTokenAuthenticatedURL.java:371)
at org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:879)
at org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:874)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709) This is very urgent. Can anyone please help.
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache Hive
01-29-2017
08:23 AM
@Pranay Vyas I have analyzed the log and found that select query from any table getting authentication error (Authentication failed, status: 403, message: Forbidden). Apart from select query all other queries are working. I am not sure why I am getting authentication error for select query. Select query is able to get the result set but getting this issue while iterating through result set. Please help.
... View more
01-29-2017
08:22 AM
@Ed Berezitsky Thanks for your reply. I have analyzed the log and found that select query from any table getting authentication error (Authentication failed, status: 403, message: Forbidden). Apart from select query all other queries are working. I am not sure why I am getting authentication error for select query. Select query is able to get the result set but getting this issue while iterating through result set. Please help.
... View more
01-27-2017
04:58 AM
@Pranay Vyas This is kerberized cluster. And I am using hive principal and keytab for authentication. All create table and insert data query are working from java api. But for any select query I am getting this authentication error (Authentication failed, status: 403, message: Forbidden). Please help.
... View more
01-24-2017
06:53 AM
@Pranay Vyas Thanks for your quick response. Hive table created successfully by hive CLI . below is the table structure. hive> desc teradata_demo_table;
OK
id int
col_date string
col_time string
col_timestamp string
col_timestamp_tz string
col_time_tz string
col_interval_yr_month string
col_interval_yr string
col_interval_month string
col_interval_day string
col_interval_day_hr string
col_period_date string
col_period_timestamp string
col_byteint tinyint
col_smallint smallint
col_integer int
col_bigint bigint
col_decimal decimal(8,2)
col_float string
col_char char(7)
col_varchar varchar(16384)
Time taken: 0.379 seconds, Fetched: 21 row(s) Full Log: 2017-01-24 16:01:38,840 INFO [HiveServer2-Background-Pool: Thread-10098]: exec.Utilities (Utilities.java:getBaseWork(400)) - PLAN PATH = hdfs://ctsc00691239901.cts.com:8020/tmp/hive/hive/ae98860b-bdbe-45a1-9212-172a3bae152b/hive_2017-01-24_16-01-38_673_8353564872225299216-27/-mr-10003/0cfdbbd9-00ba-4f02-a100-5287bdaedfb8/reduce.xml
2017-01-24 16:01:38,841 INFO [HiveServer2-Background-Pool: Thread-10098]: exec.Utilities (Utilities.java:getBaseWork(466)) - File not found: File does not exist: /tmp/hive/hive/ae98860b-bdbe-45a1-9212-172a3bae152b/hive_2017-01-24_16-01-38_673_8353564872225299216-27/-mr-10003/0cfdbbd9-00ba-4f02-a100-5287bdaedfb8/reduce.xml
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71)
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61)
...
2017-01-24 16:01:38,841 INFO [HiveServer2-Background-Pool: Thread-10098]: exec.Utilities (Utilities.java:getBaseWork(467)) - No plan file found: hdfs://ctsc00691239901.cts.com:8020/tmp/hive/hive/ae98860b-bdbe-45a1-9212-172a3bae152b/hive_2017-01-24_16-01-38_673_8353564872225299216-27/-mr-10003/0cfdbbd9-00ba-4f02-a100-5287bdaedfb8/reduce.xml
2017-01-24 16:01:38,848 INFO [HiveServer2-Background-Pool: Thread-10098]: hdfs.DFSClient (DFSClient.java:getDelegationToken(1047)) - Created HDFS_DELEGATION_TOKEN token 3924 for hive on 10.223.72.129:8020
2017-01-24 16:01:38,857 INFO [HiveServer2-Background-Pool: Thread-10098]: mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(249)) - Cleaning up the staging area /user/hive/.staging/job_1484412561270_0059
2017-01-24 16:01:38,858 ERROR [HiveServer2-Background-Pool: Thread-10098]: exec.Task (SessionState.java:printError(962)) - Job Submission failed with exception 'java.io.IOException(java.lang.reflect.UndeclaredThrowableException)'
java.io.IOException: java.lang.reflect.UndeclaredThrowableException
at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:892)
at org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:86)
at org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2291)
...
Caused by: java.lang.reflect.UndeclaredThrowableException
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1727)
at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:874)
... 40 more
Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, status: 403, message: Forbidden
at org.apache.hadoop.security.authentication.client.AuthenticatedURL.extractToken(AuthenticatedURL.java:274)
at org.apache.hadoop.security.authentication.client.PseudoAuthenticator.authenticate(PseudoAuthenticator.java:77)
at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:128)
at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:214)
...
2017-01-24 16:01:38,860 INFO [ATS Logger 0]: hooks.ATSHook (ATSHook.java:createPostHookEvent(193)) - Received post-hook notification for :hive_20170124160138_1cd543d0-afe0-46ee-95c6-b9fc1d4440f6
2017-01-24 16:01:38,860 ERROR [HiveServer2-Background-Pool: Thread-10098]: ql.Driver (SessionState.java:printError(962)) - FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
...
2017-01-24 16:01:38,861 ERROR [HiveServer2-Background-Pool: Thread-10098]: operation.Operation (SQLOperation.java:run(209)) - Error running hive query:
org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:315)
at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:156)
at org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:71)
at org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:206)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
at org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
All other commands are running fine. This is simple hive query where I want to store the query result in local directory. Please help.
... View more
01-20-2017
09:31 AM
I am trying to execute below query using java api INSERT OVERWRITE LOCAL DIRECTORY '/home/bigframe/aps/temp' select * from teradata_demo_table And I am getting below error java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:296)
at com.bigframe.perl.api.HiveReconTableCreate.doHiveTableCreation(HiveReconTableCreate.java:82)
at com.bigframe.perl.api.HiveReconTableCreate.main(HiveReconTableCreate.java:142) Same query is working from Hive CLI. Also "select * from teradata_demo_table" is working from java API. Please suggest.
... View more
Labels:
- Labels:
-
Apache Hive
01-10-2017
05:12 AM
@Nicola Poffo Can you please suggest how to start? did you read any book?
... View more
11-22-2016
05:49 AM
@Greg Keys Thanks for your response. As per sqoop document --validate option does not work with Hive and HBase. What strategy should I take for Hive and HBase ? Please suggest.
... View more
11-17-2016
10:02 AM
1 Kudo
I am using sqoop for data transfer from RDBMS to HDFS, hive and HBase. How can I validate the data after sqoop command execution in HDFS, Hive and HBase? Please suggest. What will be the best strategy for data validation ?
... View more
Labels: