Reply
Highlighted
Vid
New Contributor
Posts: 5
Registered: ‎02-07-2018

Hue Metastore manager doesn't return any sample data

Updated hive.fetch.task.conversion value to none from more. the reason for this change was my organisation wants to run all the queries as mapreduce jobs.

After the restart of hive services, hive tables within the metastore manager doesn't return any sample data.

I'm getting results If i run a select statement on the same table through hive query editor.

Is it some sort configuration that i'm missing here?

 

Thanks

Vid 

Cloudera Employee
Posts: 266
Registered: ‎03-23-2015

Re: Hue Metastore manager doesn't return any sample data

Hi Vid,

Force every query to run as MR job will affect the way Hue fetches sample data, as it will take longer time to return.

What's the reason to force every query to run as MR job? hive.fetch.task.conversion is there for a reason to improve certain query performance.
Vid
New Contributor
Posts: 5
Registered: ‎02-07-2018

Re: Hue Metastore manager doesn't return any sample data

The reason for changing the hive.fetch.task.conversion to none was because of users losing permissions to random hive tables when we run simple queries in local mode. when we force the queries to run as mapreduce jobs the issue disappears. Sounds fishy for us.

 

Cloudera Employee
Posts: 266
Registered: ‎03-23-2015

Re: Hue Metastore manager doesn't return any sample data

Thanks for the update Vid.

Then I think we should focus on fixing the issue that users keep losing permission to random hive tables. Changing hive.fetch.task.conversion is not the correct way to hide the issue.

Can you please describe the permission issue in more detail?

- When does it happen? How often?
- Any workflow might trigger such problem?
- Does it happen to all users?
- When it happens, what error do you see?
- How do you create those tables that have issues? Do you create through Hive, Impala, Spark or other thirdparty tools?

Thanks
Vid
New Contributor
Posts: 5
Registered: ‎02-07-2018

Re: Hue Metastore manager doesn't return any sample data

[ Edited ]

- IT happened on last december and again on january in this year.

 

 

- we Just use hive to query the database, so i dont believe there will be any work flow to cause this issue

 

 

- yes it happens to all the users on different tables in different database. If one user is losing permission on the tables, the table gets locked resulting no one can access it

 

 

Error message is:

 Bad status for request TFetchResultsReq(fetchType=0, operationHandle=TOperationHandle(hasResultSet=True, modifiedRowCount=None, operationType=0, operationId=THandleIdentifier(secret='\xe1l\xf9Z\x89\xc2C\xf7\xac!V\x7f:\xb2\xb3\xc1', guid='i\xa9\x08\x11\xba\xd9Dj\x84\xc5\xd3\xe2\t\x9d\xe07')), orientation=4, maxRows=100): TFetchResultsResp(status=TStatus(errorCode=0, errorMessage='java.io.IOException: java.lang.RuntimeException: serious problem', sqlState=None, infoMessages=['*org.apache.hive.service.cli.HiveSQLException:java.io.IOException: java.lang.RuntimeException: serious problem:25:24', 'org.apache.hive.service.cli.operation.SQLOperation:getNextRowSet:SQLOperation.java:338', 'org.apache.hive.service.cli.operation.OperationManager:getOperationNextRowSet:OperationManager.java:223', 'org.apache.hive.service.cli.session.HiveSessionImpl:fetchResults:HiveSessionImpl.java:714', 'sun.reflect.GeneratedMethodAccessor44:invoke::-1', 'sun.reflect.DelegatingMethodAccessorImpl:invoke:DelegatingMethodAccessorImpl.java:43', 'java.lang.reflect.Method:invoke:Method.java:498', 'org.apache.hive.service.cli.session.HiveSessionProxy:invoke:HiveSessionProxy.java:78', 'org.apache.hive.service.cli.session.HiveSessionProxy:access$000:HiveSessionProxy.java:36', 'org.apache.hive.service.cli.session.HiveSessionProxy$1:run:HiveSessionProxy.java:63', 'java.security.AccessController:doPrivileged:AccessController.java:-2', 'javax.security.auth.Subject:doAs:Subject.java:422', 'org.apache.hadoop.security.UserGroupInformation:doAs:UserGroupInformation.java:1657', 'org.apache.hive.service.cli.session.HiveSessionProxy:invoke:HiveSessionProxy.java:59', 'com.sun.proxy.$Proxy36:fetchResults::-1', 'org.apache.hive.service.cli.CLIService:fetchResults:CLIService.java:455', 'org.apache.hive.service.cli.thrift.ThriftCLIService:FetchResults:ThriftCLIService.java:676', 'org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults:getResult:TCLIService.java:1553', 'org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults:getResult:TCLIService.java:1538','org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow:FetchOperator:java:445','*java.util.concurrent.ExecutionException:org.apache.hadoop.security.AccessControlException:permission denied: user=xxxx, access=READ, path="/xx/xx":35:2'

 

- The issue happens for random tables, and we use almost the same procedure for all the table creation.

Cloudera Employee
Posts: 266
Registered: ‎03-23-2015

Re: Hue Metastore manager doesn't return any sample data

I am curious to see what are the error message from HS2 log when Hue returns such errors when fetchResults, can you please check if there is any errors on the HS2 log?
Vid
New Contributor
Posts: 5
Registered: ‎02-07-2018

Re: Hue Metastore manager doesn't return any sample data

[ Edited ]

When i had a look at the logs it says Permission issue but i can confirm that the user has full control permissions on the database. The full error message from HS2 log:

 

 WARN     - [HiveServer2-Handler-Pool: Thread-137044] - thrift.ThriftCLIService - Error fetching results:
org.apache.hive.service.cli.HiveSQLException: java.io.IOException: java.lang.RuntimeException: serious problem
        at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:338)
        at org.apache.hive.service.cli.operation.OperationManager.getOperationNextRowSet(OperationManager.java:223)
        at org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:714)
        at sun.reflect.GeneratedMethodAccessor44.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
        at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
        at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
        at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
        at com.sun.proxy.$Proxy36.fetchResults(Unknown Source)
        at org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:455)
        at org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:676)
        at org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1553)
        at org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1538)
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: java.lang.RuntimeException: serious problem
        at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:507)
        at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:414)
        at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:140)
        at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1690)
        at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:333)
        ... 24 more
Caused by: java.lang.RuntimeException: serious problem
        at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1022)
        at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1049)
        at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextSplits(FetchOperator.java:362)
        at org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:294)
        at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:445)
        ... 28 more
Caused by: java.util.concurrent.ExecutionException: org.apache.hadoop.security.AccessControlException: Permission denied: user=XXXX, access=READ, path="/apps/hive/warehouse/XXXX"
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:192)
        at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:999)
        ... 32 more
Caused by: org.apache.hadoop.security.AccessControlException: Permission denied: user=XXXX, access=READ, path="/apps/hive/warehouse/XXXX"
        at sun.reflect.GeneratedConstructorAccessor219.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
        at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
        at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:2088)
 at org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.<init>(DistributedFileSystem.java:944)
        at org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.<init>(DistributedFileSystem.java:927)
        at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:872)
        at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:868)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.listLocatedStatus(DistributedFileSystem.java:886)
        at org.apache.hadoop.fs.FileSystem.listLocatedStatus(FileSystem.java:1694)
        at org.apache.hadoop.hive.shims.Hadoop23Shims.listLocatedStatus(Hadoop23Shims.java:669)
        at org.apache.hadoop.hive.ql.io.AcidUtils.getAcidState(AcidUtils.java:361)
        at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.call(OrcInputFormat.java:635)
        at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$FileGenerator.call(OrcInputFormat.java:621)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        ... 3 more
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=XXXX, access=READ, path="/apps/hive/warehouse/XXXX"
        at org.apache.hadoop.ipc.Client.call(Client.java:1475)
        at org.apache.hadoop.ipc.Client.call(Client.java:1412)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
        at com.sun.proxy.$Proxy10.getListing(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getListing(ClientNamenodeProtocolTranslatorPB.java:573)
        at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at com.sun.proxy.$Proxy11.getListing(Unknown Source)
        at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:2086)

Cloudera Employee
Posts: 266
Registered: ‎03-23-2015

Re: Hue Metastore manager doesn't return any sample data

Do you have ACLs for /apps/hive/warehouse/XXXX? What's the output for:

hdfs dfs -getfacl /apps/hive/warehouse/XXXX

and

hdfs dfs -ls -d /apps/hive/warehouse/XXXX
Vid
New Contributor
Posts: 5
Registered: ‎02-07-2018

Re: Hue Metastore manager doesn't return any sample data

yes, we do have ACL's for hive databases

 

Owner and group are the same for all the databases but when querying a few databases we get the error message

 

Output for hdfs dfs -getfacl /apps/hive/warehouse/XXXX:

# file: /apps/hive/warehouse/XXXX
# owner: hive-sys-account
# group: hadoop-group
getfacl: Isilon doesn't yet support HDFS ACLs.

 

Output for  hdfs dfs -ls -d /apps/hive/warehouse/XXXX :

drwxrwx--- -     hive-sys-account    hadoop-group   0   2018-04-05 16:48   /apps/hive/warehouse/XXXX

Announcements