Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

TezReflectionException after changing doas to false for llap and hive... .only occurs through zeppelin/livy

TezReflectionException after changing doas to false for llap and hive... .only occurs through zeppelin/livy

Expert Contributor

I need to track user access to files/tables and resource management at a per user level. I tried using hive & llap with "do as user". This means they are impersonating. This is great as it tracks in Ranger who did what. I have an issue now. Zeppelin & Livy started puking errors ever since I made the change with:

I can run a simply query in Zeppelin (w/ LIvy) and get the below error same query runs fine via jdbc connector.

java.sql.SQLException: java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, vertexId=vertex_1522011783190_0104_1_00, diagnostics=[Vertex vertex_1522011783190_0104_1_00 [Map 1] killed/failed due to:INIT_FAILURE, Fail to create InputInitializerManager, org.apache.tez.dag.api.TezReflectionException: Unable to instantiate class with 1 arguments: org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator
	at org.apache.tez.common.ReflectionUtils.getNewInstance(ReflectionUtils.java:71)
	at org.apache.tez.common.ReflectionUtils.createClazzInstance(ReflectionUtils.java:89)
	at org.apache.tez.dag.app.dag.RootInputInitializerManager$1.run(RootInputInitializerManager.java:152)
	at org.apache.tez.dag.app.dag.RootInputInitializerManager$1.run(RootInputInitializerManager.java:148)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
	at org.apache.tez.dag.app.dag.RootInputInitializerManager.createInitializer(RootInputInitializerManager.java:148)
	at org.apache.tez.dag.app.dag.RootInputInitializerManager.runInputInitializers(RootInputInitializerManager.java:121)
	at org.apache.tez.dag.app.dag.impl.VertexImpl.setupInputInitializerManager(VertexImpl.java:4031)
	at org.apache.tez.dag.app.dag.impl.VertexImpl.access$3100(VertexImpl.java:204)
	at org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.handleInitEvent(VertexImpl.java:2855)
	at org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.transition(VertexImpl.java:2802)
	at org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.transition(VertexImpl.java:2784)
	at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
	at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
	at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
	at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
	at org.apache.tez.state.StateMachineTez.doTransition(StateMachineTez.java:59)
	at org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:1925)
	at org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:203)
	at org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:2291)
	at org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:2277)
	at org.apache.tez.common.AsyncDispatcher.dispatch(AsyncDispatcher.java:184)
	at org.apache.tez.common.AsyncDispatcher$1.run(AsyncDispatcher.java:115)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.tez.common.ReflectionUtils.getNewInstance(ReflectionUtils.java:68)
	... 25 more
Caused by: java.lang.IllegalArgumentException: No running LLAP daemons! Please check LLAP service status and zookeeper configuration
	at com.google.common.base.Preconditions.checkArgument(Preconditions.java:92)
	at org.apache.hadoop.hive.ql.exec.tez.Utils.getSplitLocationProvider(Utils.java:47)
	at org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.<init>(HiveSplitGenerator.java:121)
	... 30 more
]Vertex killed, vertexName=Reducer 2, vertexId=vertex_1522011783190_0104_1_01, diagnostics=[Vertex received Kill in NEW state., Vertex vertex_1522011783190_0104_1_01 [Reducer 2] killed/failed due to:OTHER_VERTEX_FAILURE]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:1
 

I assume this is permission issue but I'm not sure how to get down to the permission issue. The application isn't availble for me to check logs, and tez leaves behine a zombie like session.

Any ideas on how to get more information to fix this?

Don't have an account?
Coming from Hortonworks? Activate your account here