Support Questions
Find answers, ask questions, and share your expertise

Zeppelin notebook with spark interpreter impersonation failing

Zeppelin notebook with spark interpreter impersonation failing

Explorer

Hi,


This is my script:

%spark2.pyspark

import math
print("The value of pi is", math.pi)


This is the output:

org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: zeppelin is not allowed to impersonate mansop
    at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1497)
    at org.apache.hadoop.ipc.Client.call(Client.java:1443)
    at org.apache.hadoop.ipc.Client.call(Client.java:1353)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
    at com.sun.proxy.$Proxy16.getClusterMetrics(Unknown Source)
    at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getClusterMetrics(ApplicationClientProtocolPBClientImpl.java:259)
    at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)


This is my Zeppelin impersonation config:

export SPARK_HOME=/usr/hdp/current/spark2-client/
export PYTHONPATH=$SPARK_HOME/python/:$PYTHONPATH
export PYTHONPATH=$SPARK_HOME/python/lib/py4j-0.10.7-src.zip:$PYTHONPATH
export SPARK_YARN_USER_ENV="PYTHONPATH=${PYTHONPATH}"
export ZEPPELIN_IMPERSONATE_CMD='sudo -H -u ${ZEPPELIN_IMPERSONATE_USER} bash -c '
export ZEPPELIN_IMPERSONATE_SPARK_PROXY_USER=${ZEPPELIN_IMPERSONATE_USER}


This is my HDFS impersonation config:

hadoop.proxyuser.hdfs.groups = *
hadoop.proxyuser.hdfs.hosts = *
hadoop.proxyuser.hive.groups = *
hadoop.proxyuser.hive.hosts = *
hadoop.proxyuser.livy.groups = *
hadoop.proxyuser.livy.hosts = *
hadoop.proxyuser.root.groups = *
hadoop.proxyuser.root.hosts = *


What am I mising?


Thank you