Hi @satvaddi , If you are running in a Ranger RAZ enabled environment you don't need all these settings: > --conf "spark.hadoop.hadoop.security.authentication=KERBEROS" \ > --conf "spark.hadoop.hadoop.security.authorization=true" \ > --conf "spark.hadoop.fs.s3a.delegation.token.binding=org.apache.knox.gateway.cloud.idbroker.s3a.IDBDelegationTokenBinding" \ > --conf "spark.hadoop.fs.s3a.idb.auth.token.enabled=true" \ > --conf "spark.hadoop.fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider" \ > --conf "spark.hadoop.fs.s3a.security.credential.provider.path=jceks://hdfs/user/infa/knox_credentials.jceks" \ > --conf "spark.hadoop.fs.s3a.endpoint=s3.amazonaws.com" \ > --conf "spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem" \ To me it looks like you are bypassing Raz by setting this parameter: > --conf "spark.hadoop.fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider" \ This, I would check whether the instance profile (IAM Role attached to the cluster) does not have too much privileges. Like access to data. This should be controlled in Ranger instead.
... View more
