Support Questions
Find answers, ask questions, and share your expertise

Streaming s3 files using Hadoop And spark

Streaming s3 files using Hadoop And spark

I was trying to stream file from s3 bucket which is in another aws account. The way that i can access that account is through session token , access key and secret key. I am authenticating using fs.s3a.session.token, fs.s3a.access.key and fs.s3a.secret.key.

The problem that i am facing is that , once i deploy this service it works for sometimes and then I will start getting this error:

 

org.apache.hadoop.fs.s3a.AWSBadRequestException: getFileStatus on s3a://gehc-us-west-2-dl-80967290-1796-4131-a8a2-1aab359a3b92/c66bbb14-c91e-4175-9f1c-5ec658fc4cd0/00264.csv: com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: 6C3695CEF0A1E8A9; S3 Extended Request ID: xgWBNn+IpySOrnG/tw0gPJFdkXRbjHGQhqRRYpN3CZDipeRm+vU9q3B9g7sUxBDNyUbE/zMIXI8=), S3 Extended Request ID: xgWBNn+IpySOrnG/tw0gPJFdkXRbjHGQhqRRYpN3CZDipeRm+vU9q3B9g7sUxBDNyUbE/zMIXI8=:400 Bad Request: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: 6C3695CEF0A1E8A9; S3 Extended Request ID: xgWBNn+IpySOrnG/tw0gPJFdkXRbjHGQhqRRYpN3CZDipeRm+vU9q3B9g7sUxBDNyUbE/zMIXI8=)

at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:212).