FetchS3Object - Failed to retrieve S3 object error - NiFi

Question

Hello

I am having an issue retrieving bucket contents into NiFi using the FetchS3Object processor. I have configured the ListS3 processor to pull in a json array containing all the bucket information, I'm happy that's working ok... example output:

I have configured the FetchS3Object processor with the same Accesskey, Secret etc but I get the following error, I haven't been able to find much online ref this error, can anyone see where I'm going wrong?

FetchS3Object Config:

Any assistance would be greatly appreciated, I've seen a lot of questions asked about these two processors but haven't found anything about this particular error.

Cheers

Andy

Griggsy · Accepted Answer

Hello @Azhar_Shaikh1

Thanks for the reply, as it turns out it wasn't a service account problem. We found that the ListS3's output included a 'key' field, and this is what was required in The FetchS3Object processor for 'Object Key'.

So the fix I applied was to split the json into individual records (SplitJson), then pull the keys out as attributes (EvaluateJsonPath) then input ${key} into the FetchS3 processor.. worked a treat.

Azhar_Shaikh1 · Answer

Hello @Griggsy

I can see from the error (Service:Amazon S3 Error code :403, Access Denied)

The error code 403 means to be an authorization issue from cloud provider end. If you are using any service account with the processor, can you please check and confirm if you the service account have access to the bucket that you have configured in Nifi ?

Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Thanks,

Azhar

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.