question Re: Error while running hive queries from Zeppelin on HDP 2.3.2 in Support Questions

Error while running hive queries from Zeppelin on HDP 2.3.2

abajwa — Sat, 28 Nov 2015 07:06:04 GMT

Am getting error "Required field 'serverProtocolVersion' is unset" when trying to run Hive queries from recent build of Zeppelin on HDP 2.3.2/Spark 1.4.1. Tried both Zeppelin service and compiling Zeppelin. Spark and Phoenix jobs run fine. Hive used to work fine before.

Google search suggests issue may be caused by below...anyone else run into this?

https://issues.apache.org/jira/browse/HIVE-6050

http://stackoverflow.com/questions/24694415/required-field-client-protocol-is-unset

Using default proxy settings for Hive

hadoop.proxyuser.hive.hosts = *

hadoop.proxyuser.hive.groups = users

Full error

ERROR [2015-11-27 04:41:26,630] ({pool-2-thread-2} HiveConnection.java[openSession]:466) - Error opening session

org.apache.thrift.protocol.TProtocolException: Required field 'serverProtocolVersion' is unset! Struct:TOpenSessionResp(status:TStatus(statusCode:ERROR_STATUS, infoMessages:[*org.apache.hive.service.cli.HiveSQLException:Failed to open new session: java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: hive is not allowed to impersonate hive:13:12, org.apache.hive.service.cli.session.SessionManager:openSession:SessionManager.java:266, org.apache.hive.service.cli.CLIService:openSessionWithImpersonation:CLIService.java:202, org.apache.hive.service.cli.thrift.ThriftCLIService:getSessionHandle:ThriftCLIService.java:402, org.apache.hive.service.cli.thrift.ThriftCLIService:OpenSession:ThriftCLIService.java:297, org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession:getResult:TCLIService.java:1253, org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession:getResult:TCLIService.java:1238, org.apache.thrift.ProcessFunction:process:ProcessFunction.java:39, org.apache.thrift.TBaseProcessor:process:TBaseProcessor.java:39, org.apache.hive.service.auth.TSetIpAddressProcessor:process:TSetIpAddressProcessor.java:56, org.apache.thrift.server.TThreadPoolServer$WorkerProcess:run:TThreadPoolServer.java:285, java.util.concurrent.ThreadPoolExecutor:runWorker:ThreadPoolExecutor.java:1145, java.util.concurrent.ThreadPoolExecutor$Worker:run:ThreadPoolExecutor.java:615, java.lang.Thread:run:Thread.java:745, *java.lang.RuntimeException:java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: hive is not allowed to impersonate hive:21:8, org.apache.hive.service.cli.session.HiveSessionProxy:invoke:HiveSessionProxy.java:83, org.apache.hive.service.cli.session.HiveSessionProxy:access$000:HiveSessionProxy.java:36, org.apache.hive.service.cli.session.HiveSessionProxy$1:run:HiveSessionProxy.java:63, java.security.AccessController:doPrivileged:AccessController.java:-2, javax.security.auth.Subject:doAs:Subject.java:415, org.apache.hadoop.security.UserGroupInformation:doAs:UserGroupInformation.java:1657, org.apache.hive.service.cli.session.HiveSessionProxy:invoke:HiveSessionProxy.java:59, com.sun.proxy.$Proxy19:open::-1, org.apache.hive.service.cli.session.SessionManager:openSession:SessionManager.java:258, *java.lang.RuntimeException:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: hive is not allowed to impersonate hive:26:5, org.apache.hadoop.hive.ql.session.SessionState:start:SessionState.java:522, org.apache.hive.service.cli.session.HiveSessionImpl:open:HiveSessionImpl.java:137, sun.reflect.GeneratedMethodAccessor10:invoke::-1, sun.reflect.DelegatingMethodAccessorImpl:invoke:DelegatingMethodAccessorImpl.java:43, java.lang.reflect.Method:invoke:Method.java:606, org.apache.hive.service.cli.session.HiveSessionProxy:invoke:HiveSessionProxy.java:78, *org.apache.hadoop.ipc.RemoteException:User: hive is not allowed to impersonate hive:45:19, org.apache.hadoop.ipc.Client:call:Client.java:1427, org.apache.hadoop.ipc.Client:call:Client.java:1358, org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker:invoke:ProtobufRpcEngine.java:229, com.sun.proxy.$Proxy14:getFileInfo::-1, org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB:getFileInfo:ClientNamenodeProtocolTranslatorPB.java:771, sun.reflect.GeneratedMethodAccessor7:invoke::-1, sun.reflect.DelegatingMethodAccessorImpl:invoke:DelegatingMethodAccessorImpl.java:43, java.lang.reflect.Method:invoke:Method.java:606, org.apache.hadoop.io.retry.RetryInvocationHandler:invokeMethod:RetryInvocationHandler.java:187, org.apache.hadoop.io.retry.RetryInvocationHandler:invoke:RetryInvocationHandler.java:102, com.sun.proxy.$Proxy15:getFileInfo::-1, org.apache.hadoop.hdfs.DFSClient:getFileInfo:DFSClient.java:2116, org.apache.hadoop.hdfs.DistributedFileSystem$22:doCall:DistributedFileSystem.java:1305, org.apache.hadoop.hdfs.DistributedFileSystem$22:doCall:DistributedFileSystem.java:1301, org.apache.hadoop.fs.FileSystemLinkResolver:resolve:FileSystemLinkResolver.java:81, org.apache.hadoop.hdfs.DistributedFileSystem:getFileStatus:DistributedFileSystem.java:1301, org.apache.hadoop.fs.FileSystem:exists:FileSystem.java:1424, org.apache.hadoop.hive.ql.session.SessionState:createRootHDFSDir:SessionState.java:596, org.apache.hadoop.hive.ql.session.SessionState:createSessionDirs:SessionState.java:554, org.apache.hadoop.hive.ql.session.SessionState:start:SessionState.java:508], errorCode:0, errorMessage:Failed to open new session: java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: hive is not allowed to impersonate hive), serverProtocolVersion:null)

Re: Error while running hive queries from Zeppelin on HDP 2.3.2

gbraccialli3 — Sat, 28 Nov 2015 07:13:16 GMT

@Ali Bajwa

I got this error when trying to connect to spark 1.4.1 thrift server using beeline that comes with sandbox 2.3.2. Spark 1.5.2 thrift server works fine with beeline 2.3.2.

Looks like new zeppelin is using new sql client, what spark version are you using?

Re: Error while running hive queries from Zeppelin on HDP 2.3.2

abajwa — Sat, 28 Nov 2015 08:21:26 GMT

Thanks @Guilherme Braccialli yes this is with HDP 2.3.2 / Spark 1.4.1. Interesting...so using Spark 1.5.2 will fix this? I will try this out.

Re: Error while running hive queries from Zeppelin on HDP 2.3.2

gbraccialli3 — Sat, 28 Nov 2015 08:25:05 GMT

It should. Actually I'm using Spark 1.5.1 Technical Preview, but I think 1.5.2 should work also.

Re: Error while running hive queries from Zeppelin on HDP 2.3.2

nsabharwal — Sat, 28 Nov 2015 08:40:06 GMT

@Ali Bajwa Just curious ..what happens if you set

hadoop.proxyuser.hive.groups = *

Re: Error while running hive queries from Zeppelin on HDP 2.3.2

abajwa — Sun, 29 Nov 2015 10:18:04 GMT

(Adding as answer as it too big to add as comment): Ran some tests and here are the findings. Changing proxyuser helps but still having issues getting queries that result in Tez jobs working for some reason. With 2.3.0 sandbox (spark 1.3.1) all hive queries seem to run fine

Test 1:

2.3.0.0-2557 (sandbox)
HDP Spark 1.3.1
hadoop.proxyuser.hive.groups=users
Results: all hive queries works fine

Test 2:

2.3.2.0-2950
HDP Spark 1.4.1
hadoop.proxyuser.hive.groups=users
Results: Error when running any hive query: Required field 'serverProtocolVersion' is unset
running from hive cli as zeppelin user runs fine

Test 3:

Ambari installed 2.3.2.0-2950 env
HDP Spark 1.4.1
hadoop.proxyuser.hive.groups=*
hive queries that don’t generate tez job work fine
Error when running hive query that generate tez job : FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask
running from hive cli as zeppelin user runs fine

Test 4:

Ambari installed 2.3.2.0-2950 env
Installed Spark 1.5.1 TP
hadoop.proxyuser.hive.groups=users
Results: Error when running any hive query: Required field 'serverProtocolVersion' is unset
running from hive cli as zeppelin user runs fine

Test 5:

Ambari installed 2.3.2.0-2950 env
Installed Spark 1.5.1 TP
hadoop.proxyuser.hive.groups=*
hive queries that don’t generate tez job work fine
Error when running hive query that generate tez job : FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask
running from hive cli as zeppelin user runs fine

Re: Error while running hive queries from Zeppelin on HDP 2.3.2

nsabharwal — Sun, 29 Nov 2015 20:00:58 GMT

@vshukla

Re: Error while running hive queries from Zeppelin on HDP 2.3.2

jstraub — Mon, 30 Nov 2015 16:31:28 GMT

@Ali Bajwa saw the same error with spark 1.4.1, it did work with 1.5.1, so it looks like its fixed.

Re: Error while running hive queries from Zeppelin on HDP 2.3.2

jstraub — Mon, 30 Nov 2015 16:32:27 GMT

nice one!

Re: Error while running hive queries from Zeppelin on HDP 2.3.2

vjain — Fri, 11 Dec 2015 10:04:52 GMT

hadoop.proxyuser.hive.groups = *

Worked for me. Thanks @Ali Bajwa

Re: Error while running hive queries from Zeppelin on HDP 2.3.2

aervits — Thu, 11 Feb 2016 05:25:10 GMT

@Ali Bajwa is this still an issue with the latest Spark and Sandbox? I have a user with exact same issues and no means to fix it yet. He claimed he tried fixing this using this article.