Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Streaming s3 files using Hadoop And spark

Streaming s3 files using Hadoop And spark

I was trying to stream file from s3 bucket which is in another aws account. The way that i can access that account is through session token , access key and secret key. I am authenticating using fs.s3a.session.token, fs.s3a.access.key and fs.s3a.secret.key.

The problem that i am facing is that , once i deploy this service it works for sometimes and then I will start getting this error:

 

org.apache.hadoop.fs.s3a.AWSBadRequestException: getFileStatus on s3a://gehc-us-west-2-dl-80967290-1796-4131-a8a2-1aab359a3b92/c66bbb14-c91e-4175-9f1c-5ec658fc4cd0/00264.csv: com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: 6C3695CEF0A1E8A9; S3 Extended Request ID: xgWBNn+IpySOrnG/tw0gPJFdkXRbjHGQhqRRYpN3CZDipeRm+vU9q3B9g7sUxBDNyUbE/zMIXI8=), S3 Extended Request ID: xgWBNn+IpySOrnG/tw0gPJFdkXRbjHGQhqRRYpN3CZDipeRm+vU9q3B9g7sUxBDNyUbE/zMIXI8=:400 Bad Request: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: 6C3695CEF0A1E8A9; S3 Extended Request ID: xgWBNn+IpySOrnG/tw0gPJFdkXRbjHGQhqRRYpN3CZDipeRm+vU9q3B9g7sUxBDNyUbE/zMIXI8=)

at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:212).

 

Don't have an account?
Coming from Hortonworks? Activate your account here