question Re: How to restrict valid users from submitting Spark YARN Job? in Archives of Support Questions (Read Only)

How to restrict valid users from submitting Spark YARN Job?

smartninja723 — Wed, 18 May 2016 00:12:52 GMT

Hi Guys,

On our Kerberized HDP, I tested that a valid A/D user once granted the TGT using kinit, is able to submit the spark job (using spark shell and also using spark-submit). However, I would like to restrict a few groups and users from submitting the job to the cluster. Is there a way around?

I see in the documentation : http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_spark-guide/content/installing-kerb-spark.html which talks about creating a separate user : spark and keytab for it, and spark user will submit the jobs. ( Personally I don't like the idea to submit all the jobs by a single user)

Thanks.

Re: How to restrict valid users from submitting Spark YARN Job?

sball — Wed, 18 May 2016 00:29:26 GMT

You can do this with YARN ACLs against the relevant yarn queues. A better way of implementing this would be to use Ranger to set YARN access controls to a given set of queues. You would then require spark users to add queue to their submit job, and the authentication would be enforced. To ensure this, prevent user access to the default queue.

Re: How to restrict valid users from submitting Spark YARN Job?

smartninja723 — Wed, 18 May 2016 16:55:25 GMT

Thanks @Simon Elliston Ball I will try that

BTW:

Re: How to restrict valid users from submitting Spark YARN Job?

sluangsay — Thu, 19 May 2016 17:23:43 GMT

This doc is just showing you an example.

Instead of using the principal "spark/blue1@EXAMPLE.COM", we could also consider using the principal "app1/blue1@EXAMPLE.COM" for one application, then using "app2/blue1@EXAMPLE.COM" for a second application etc.