Support Questions

Find answers, ask questions, and share your expertise

User impersonation in Apache Spark2 Thrift Server

Expert Contributor

Hi,

On HDP 2.6, I have configured Spark Thrift Server for Spark 1.6.x based on community wiki with which I see the queries are executed as the end user, User when connects to Spark Thrift Server for Spark-1, using beeline, I see a new instance of the process getting listed under Resource Manager with end user ( YARN Process runs as end user)

Now I am trying to configure Spark2 Thrift Server, following the official documentation .

  • Added hive.server2.enable.doAs=true
  • Added spark.jars to classpath (dataNucleus jars)
  • Set the spark.master to local
  • Restarted Spark2 Thrift Server

In my understanding Now,

  1. Queries should run as "end user" (Queries are still running as hive user)
  2. Spark2 Thrift Server when connected with spark.master=local, should be listed under Resource Manager UI with "end user" ( I do not see it listed now)
  3. When all the JDBC connections to STS are closed, STS application should disappear. As the STS is started in local mode and for each user/connection if not shred, queries are executed by the Spark Application Master launched on behalf of the end user.

Above all are not respected in Spark2 Thrift Server ( But with impersonation support in Spark-1 Thrift Server, above all three are working as expected).

Attaching the screenshots for anomalies.

I am not sure if I missed something here.

1 Queries still run as Hive.

17505-1-queries-run-as-hive.png

2. STS is not listed under Resource Manager

17506-2-sts-not-listed-in-rm.png

3. Spark2 Thrift Server Still Runs As Hive User

17507-sts-runs-as-hive-user.png

Thanks in advance.

Regards,

SS

Any inputs? @cdraper, @amcbarnett, @Ana Gillan ?

1 ACCEPTED SOLUTION

@Smart Solutions

Impersonation in Spark2 thrift is not supported yet. This could probably come in future releases.

View solution in original post

8 REPLIES 8

@Smart Solutions

Impersonation in Spark2 thrift is not supported yet. This could probably come in future releases.

Expert Contributor

@Sandeep NemuriThank you for the confirmation. When do we expect this fix?

@Smart Solutions As of now there are no timelines. Keep an eye on future releases.

New Contributor

Still not working with up to Version HDP 2.6.4.... 😞 Any updates about this? Unfortunatelly there are many Spark Jira's open related to this topics. For example this one:

https://issues.apache.org/jira/browse/SPARK-21918

Regards, Tom

New Contributor

Do we have any updates on this . I am seeing the same behaviour on HDP 2.6.5 also.

New Contributor

Any updates? please

New Contributor

I am using HDP 3.1.0 still facing issue with Spark2 Thrift server user impersonation any update on this?

New Contributor

apache kyuubi is good work

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.