Support Questions
Find answers, ask questions, and share your expertise

User impersonation in Apache Spark2 Thrift Server

Solved Go to solution

User impersonation in Apache Spark2 Thrift Server

Expert Contributor

Hi,

On HDP 2.6, I have configured Spark Thrift Server for Spark 1.6.x based on community wiki with which I see the queries are executed as the end user, User when connects to Spark Thrift Server for Spark-1, using beeline, I see a new instance of the process getting listed under Resource Manager with end user ( YARN Process runs as end user)

Now I am trying to configure Spark2 Thrift Server, following the official documentation .

  • Added hive.server2.enable.doAs=true
  • Added spark.jars to classpath (dataNucleus jars)
  • Set the spark.master to local
  • Restarted Spark2 Thrift Server

In my understanding Now,

  1. Queries should run as "end user" (Queries are still running as hive user)
  2. Spark2 Thrift Server when connected with spark.master=local, should be listed under Resource Manager UI with "end user" ( I do not see it listed now)
  3. When all the JDBC connections to STS are closed, STS application should disappear. As the STS is started in local mode and for each user/connection if not shred, queries are executed by the Spark Application Master launched on behalf of the end user.

Above all are not respected in Spark2 Thrift Server ( But with impersonation support in Spark-1 Thrift Server, above all three are working as expected).

Attaching the screenshots for anomalies.

I am not sure if I missed something here.

1 Queries still run as Hive.

17505-1-queries-run-as-hive.png

2. STS is not listed under Resource Manager

17506-2-sts-not-listed-in-rm.png

3. Spark2 Thrift Server Still Runs As Hive User

17507-sts-runs-as-hive-user.png

Thanks in advance.

Regards,

SS

Any inputs? @cdraper, @amcbarnett, @Ana Gillan ?

1 ACCEPTED SOLUTION

Accepted Solutions

Re: User impersonation in Apache Spark2 Thrift Server

@Smart Solutions

Impersonation in Spark2 thrift is not supported yet. This could probably come in future releases.

View solution in original post

7 REPLIES 7

Re: User impersonation in Apache Spark2 Thrift Server

@Smart Solutions

Impersonation in Spark2 thrift is not supported yet. This could probably come in future releases.

View solution in original post

Re: User impersonation in Apache Spark2 Thrift Server

Expert Contributor

@Sandeep NemuriThank you for the confirmation. When do we expect this fix?

Re: User impersonation in Apache Spark2 Thrift Server

@Smart Solutions As of now there are no timelines. Keep an eye on future releases.

Re: User impersonation in Apache Spark2 Thrift Server

New Contributor

Still not working with up to Version HDP 2.6.4.... :-( Any updates about this? Unfortunatelly there are many Spark Jira's open related to this topics. For example this one:

https://issues.apache.org/jira/browse/SPARK-21918

Regards, Tom

Re: User impersonation in Apache Spark2 Thrift Server

New Contributor

Do we have any updates on this . I am seeing the same behaviour on HDP 2.6.5 also.

Re: User impersonation in Apache Spark2 Thrift Server

New Contributor

Any updates? please

Re: User impersonation in Apache Spark2 Thrift Server

New Contributor

I am using HDP 3.1.0 still facing issue with Spark2 Thrift server user impersonation any update on this?