question Re: User impersonation in Apache Spark2 Thrift Server in Archives of Support Questions (Read Only)

User impersonation in Apache Spark2 Thrift Server

smartninja723 — Sun, 18 Aug 2019 03:09:33 GMT

Hi,

On HDP 2.6, I have configured Spark Thrift Server for Spark 1.6.x based on community wiki with which I see the queries are executed as the end user, User when connects to Spark Thrift Server for Spark-1, using beeline, I see a new instance of the process getting listed under Resource Manager with end user ( YARN Process runs as end user)

Now I am trying to configure Spark2 Thrift Server, following the official documentation .

Added hive.server2.enable.doAs=true
Added spark.jars to classpath (dataNucleus jars)
Set the spark.master to local
Restarted Spark2 Thrift Server

In my understanding Now,

Queries should run as "end user" (Queries are still running as hive user)
Spark2 Thrift Server when connected with spark.master=local, should be listed under Resource Manager UI with "end user" ( I do not see it listed now)
When all the JDBC connections to STS are closed, STS application should disappear. As the STS is started in local mode and for each user/connection if not shred, queries are executed by the Spark Application Master launched on behalf of the end user.

Above all are not respected in Spark2 Thrift Server ( But with impersonation support in Spark-1 Thrift Server, above all three are working as expected).

Attaching the screenshots for anomalies.

I am not sure if I missed something here.

1 Queries still run as Hive.

2. STS is not listed under Resource Manager

3. Spark2 Thrift Server Still Runs As Hive User

Thanks in advance.

Regards,

Any inputs? @cdraper, @amcbarnett, @Ana Gillan ?

Re: User impersonation in Apache Spark2 Thrift Server

sandyy006 — Fri, 30 Jun 2017 02:21:32 GMT

@Smart Solutions

Impersonation in Spark2 thrift is not supported yet. This could probably come in future releases.

Re: User impersonation in Apache Spark2 Thrift Server

smartninja723 — Fri, 30 Jun 2017 17:32:48 GMT

@Sandeep NemuriThank you for the confirmation. When do we expect this fix?

Re: User impersonation in Apache Spark2 Thrift Server

sandyy006 — Fri, 30 Jun 2017 17:38:55 GMT

@Smart Solutions As of now there are no timelines. Keep an eye on future releases.

Re: User impersonation in Apache Spark2 Thrift Server

Tom80 — Tue, 26 Jun 2018 17:39:50 GMT

Still not working with up to Version HDP 2.6.4.... 😞 Any updates about this? Unfortunatelly there are many Spark Jira's open related to this topics. For example this one:

https://issues.apache.org/jira/browse/SPARK-21918

Regards, Tom

Re: User impersonation in Apache Spark2 Thrift Server

lawang_mishra — Wed, 11 Jul 2018 22:10:45 GMT

Do we have any updates on this . I am seeing the same behaviour on HDP 2.6.5 also.

Re: User impersonation in Apache Spark2 Thrift Server

pkasinathan — Fri, 30 Nov 2018 14:57:02 GMT

Any updates? please

Re: User impersonation in Apache Spark2 Thrift Server

Ashfaque79 — Tue, 24 Sep 2019 09:41:26 GMT

I am using HDP 3.1.0 still facing issue with Spark2 Thrift server user impersonation any update on this?

Re: User impersonation in Apache Spark2 Thrift Server

just-a-little — Tue, 15 Feb 2022 03:00:35 GMT

apache kyuubi is good work