Support Questions

Find answers, ask questions, and share your expertise

spark read from different account s3 and write to my account s3

New Contributor

I have spark job which needs to read the data from s3 which is in other account**(Data Account)** and process that data. once its processed it should write back to s3 which is in my account. So I configured access and secret key of **"Data account"** like below in my spark session

val hadoopConf=sc.hadoopConfiguration 



hadoopConf.set("fs.s3a.endpoint", "") 

System.setProperty("", "true") 

val df ="s3a://DataAccountS3/path") /* Reading is success */ 


with this reading is fine, but I am getting below error when writing Status Code: 301, AWS Service: Amazon S3, AWS Request ID: A5E574113745D6A0, AWS Error Code: PermanentRedirect, AWS Error Message: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.
if I use s3n  


then getting below error Permission denied: s3n://myaccountS3/test

but If I dont configure details of Data Account and try to write some dummy data to my s3 from spark it works. So how should I configure to make both reading from different account s3 and writing to my account s3 works


@Indra s: with the S3A connector you can use per-bucket configuration options to set a different username/pass for the remote bucket



Then when you read or write s3a://myaccounts3/ then these specific username/passwords are used. For other S3A buckets, the default ones are picked up:



Please switch to using the s3a:// connector everywhere: its got much better performance and functionality than the older S3N one, which has recently been removed entirely.

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.