Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to make Spark or any other components properties non overridden

Highlighted

How to make Spark or any other components properties non overridden

Guru

Team,

Is there anyway where we can restrict users not to override few properties of spark or any other components at runtime ?Thanks in advance.

6 REPLIES 6
Highlighted

Re: How to make Spark or any other components properties non overridden

Super Mentor

@Saurabh

I do not think there is an option to restrict users to allow editing only few specific properties not others.

But if you are managing your cluster using ambari then you can have a "read Only" user get created so that that particular user will not be able to make any config changes:

As per [1] Cluster User: Users assigned to the Cluster User role are able to view information about the cluster and its services, including configurations, service status and health alerts. Effectively, this is a “read-only” user.

[1] https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.0.1/bk_ambari-administration/content/cluster_ro...
Highlighted

Re: How to make Spark or any other components properties non overridden

Guru

@Jay SenSharma Thanks for your quick response.

Can you please help me to clear following points:

  1. If I would set final to a property then would it not do the same ?
  2. Does adding a user to cluster user role prevent users not to change property at runtime in CLI.

for example if user is submitting spark job and he is giving value for spark.driver.maxResultSize=20G then can a user part of cluster user will not be able to change it.

spark-submit --verbose --master yarn --conf spark.yarn.executor.memoryOverhead=600 --num-executors 50 --conf spark.executor.memory=30G --conf spark.yarn.queue=batch --conf spark.driver.memory=30G --conf spark.kryoserializer.buffer.max=512M --conf spark.driver.cores=6 --conf spark.executor.cores=6 --conf spark.executor.instances=6 --conf spark.driver.maxResultSize=20G --conf spark.shuffle.memory.fraction=0.6 --conf spark.storage.memory.fraction=0.4 --conf spark.sql.shuffle.partitions=500 --conf "spark.executor.extraJavaOptions= ......

Highlighted

Re: How to make Spark or any other components properties non overridden

Super Mentor

@Saurabh

1. Component / Service Administrators typically define parameters as final in configuration files like core-site.xml for values that user/client applications may not alter.

The hadoop cluster "*.xml" config files can be marked as "final", indicating that they should not be overwritten by a client's configuration. It does not talk about user roles. Like which client can override it and which client can not. Once a property is declared as final then it can not be overwritten.

.

2. Ambari cluster roles are different. That only takes care of how *ambari* users will interact with the ambari using UI/Ambari Rest API. It can not be compared with the "final" feature of individual components.

Re: How to make Spark or any other components properties non overridden

Guru

@Jay SenSharma Thanks, so you mean we can not prevent user not to override property at runtime ?

Highlighted

Re: How to make Spark or any other components properties non overridden

Super Mentor

@Saurabh

I mean to say that we "can" prevent users from overriding properties at runtime using "final". (that is correct).

But i do not see an option using which a service component "userA" can do the same but not "userB". When Service Administrator mark the property as "final" then it is applied to all the users.

Highlighted

Re: How to make Spark or any other components properties non overridden

Super Mentor

@Saurabh

I mean to say that we "can" prevent users from overriding some properties at runtime using "final". (that is correct). As "final", indicates that they should not be overwritten by a client's configuration.

But i do not see an option using which a service component "userA" can make a runtime property change but not "userB". Becaue when Service Administrator mark the property as "final" then it is applied to all the users.

https://hadoop.apache.org/docs/r2.7.2/api/org/apache/hadoop/conf/Configuration.html

.

For other components ... Ranger can allow authorization, to have more control over what a user is authorized to do.

https://hortonworks.com/apache/ranger/

Don't have an account?
Coming from Hortonworks? Activate your account here