Hi,
On HDP 2.6, I have configured Spark Thrift Server for Spark 1.6.x based on community wiki with which I see the queries are executed as the end user, User when connects to Spark Thrift Server for Spark-1, using beeline, I see a new instance of the process getting listed under Resource Manager with end user ( YARN Process runs as end user)
Now I am trying to configure Spark2 Thrift Server, following the official documentation .
- Added hive.server2.enable.doAs=true
- Added spark.jars to classpath (dataNucleus jars)
- Set the spark.master to local
- Restarted Spark2 Thrift Server
In my understanding Now,
- Queries should run as "end user" (Queries are still running as hive user)
- Spark2 Thrift Server when connected with spark.master=local, should be listed under Resource Manager UI with "end user" ( I do not see it listed now)
- When all the JDBC connections to STS are closed, STS application should disappear. As the STS is started in local mode and for each user/connection if not shred, queries are executed by the Spark Application Master launched on behalf of the end user.
Above all are not respected in Spark2 Thrift Server ( But with impersonation support in Spark-1 Thrift Server, above all three are working as expected).
Attaching the screenshots for anomalies.
I am not sure if I missed something here.
1 Queries still run as Hive.
2. STS is not listed under Resource Manager
3. Spark2 Thrift Server Still Runs As Hive User
Thanks in advance.
Regards,
SS
Any inputs? @cdraper, @amcbarnett, @Ana Gillan ?
