Support Questions
Find answers, ask questions, and share your expertise

TezReflectionException after changing doas to false for llap and hive... .only occurs through zeppelin/livy

TezReflectionException after changing doas to false for llap and hive... .only occurs through zeppelin/livy

Expert Contributor

I need to track user access to files/tables and resource management at a per user level. I tried using hive & llap with "do as user". This means they are impersonating. This is great as it tracks in Ranger who did what. I have an issue now. Zeppelin & Livy started puking errors ever since I made the change with:

I can run a simply query in Zeppelin (w/ LIvy) and get the below error same query runs fine via jdbc connector.

java.sql.SQLException: java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, vertexId=vertex_1522011783190_0104_1_00, diagnostics=[Vertex vertex_1522011783190_0104_1_00 [Map 1] killed/failed due to:INIT_FAILURE, Fail to create InputInitializerManager, org.apache.tez.dag.api.TezReflectionException: Unable to instantiate class with 1 arguments: org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator
	at org.apache.tez.common.ReflectionUtils.getNewInstance(ReflectionUtils.java:71)
	at org.apache.tez.common.ReflectionUtils.createClazzInstance(ReflectionUtils.java:89)
	at org.apache.tez.dag.app.dag.RootInputInitializerManager$1.run(RootInputInitializerManager.java:152)
	at org.apache.tez.dag.app.dag.RootInputInitializerManager$1.run(RootInputInitializerManager.java:148)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
	at org.apache.tez.dag.app.dag.RootInputInitializerManager.createInitializer(RootInputInitializerManager.java:148)
	at org.apache.tez.dag.app.dag.RootInputInitializerManager.runInputInitializers(RootInputInitializerManager.java:121)
	at org.apache.tez.dag.app.dag.impl.VertexImpl.setupInputInitializerManager(VertexImpl.java:4031)
	at org.apache.tez.dag.app.dag.impl.VertexImpl.access$3100(VertexImpl.java:204)
	at org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.handleInitEvent(VertexImpl.java:2855)
	at org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.transition(VertexImpl.java:2802)
	at org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.transition(VertexImpl.java:2784)
	at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
	at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
	at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
	at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
	at org.apache.tez.state.StateMachineTez.doTransition(StateMachineTez.java:59)
	at org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:1925)
	at org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:203)
	at org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:2291)
	at org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:2277)
	at org.apache.tez.common.AsyncDispatcher.dispatch(AsyncDispatcher.java:184)
	at org.apache.tez.common.AsyncDispatcher$1.run(AsyncDispatcher.java:115)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.tez.common.ReflectionUtils.getNewInstance(ReflectionUtils.java:68)
	... 25 more
Caused by: java.lang.IllegalArgumentException: No running LLAP daemons! Please check LLAP service status and zookeeper configuration
	at com.google.common.base.Preconditions.checkArgument(Preconditions.java:92)
	at org.apache.hadoop.hive.ql.exec.tez.Utils.getSplitLocationProvider(Utils.java:47)
	at org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.<init>(HiveSplitGenerator.java:121)
	... 30 more
]Vertex killed, vertexName=Reducer 2, vertexId=vertex_1522011783190_0104_1_01, diagnostics=[Vertex received Kill in NEW state., Vertex vertex_1522011783190_0104_1_01 [Reducer 2] killed/failed due to:OTHER_VERTEX_FAILURE]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:1
 

I assume this is permission issue but I'm not sure how to get down to the permission issue. The application isn't availble for me to check logs, and tez leaves behine a zombie like session.

Any ideas on how to get more information to fix this?