question Re: Getting Error While fetching files from s3 bucket using FetchS3Object Nifi in Support Questions

Getting Error While fetching files from s3 bucket using FetchS3Object Nifi

Techie123 — Thu, 09 Feb 2023 12:06:59 GMT

Hi Team,

I have an requirement to generate temporary credentials of AWS and fetch files from the S3 bucket and use this credentials in FetchS3Object

But at FetchS3Object processor I am getting below error while fetching files from the bucket:

"FetchS3Object[id=xxx] Failed to retrieve S3 Object for FlowFile[filename=xyz.json]; routing to failure: com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain: [EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)), SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey), WebIdentityTokenCredentialsProvider: To use assume role profiles the aws-java-sdk-sts module must be on the class path., com.amazonaws.auth.profile.ProfileCredentialsProvider@3075e891: profile file cannot be null, com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper@2162bad1: Failed to connect to service endpoint: ]"

can you please help me with the issue

Re: Getting Error While fetching files from s3 bucket using FetchS3Object Nifi

steven-matison — Thu, 09 Feb 2023 13:34:44 GMT

@Techie123 You are going to need to provide credentials for the nifi calls against any s3 bucket with access controls. I would recommend working in lower nifi dev environments and use a public S3 buckets to get comfortable. This will test basics of your flow without access issues. This will also remove confusion around nifi flow functionality vs AWS access issues and help you learn when/where to use the different ways (key, or a controler service w/ credentials) to provide access from nifi to s3.

Re: Getting Error While fetching files from s3 bucket using FetchS3Object Nifi

Techie123 — Thu, 09 Feb 2023 13:38:23 GMT

I need to use the controller service with credentials but still am getting the error. You have an Idea how can i do that. Any suggestions?

Re: Getting Error While fetching files from s3 bucket using FetchS3Object Nifi

steven-matison — Thu, 09 Feb 2023 14:25:09 GMT

@Techie123 Well, like i said, you have to learn the aws side of providing access to a bucket. A public bucket starting point will show you what you have to do, inside of the bucket config, to allow other systems to access that bucket. For example starting from public open bucket, to whatever access control level you ultimately need to have. Getting lost in that space is not necessarily a "nifi" thing.... so my recommendation is to build nifi with public bucket, THEN when it works, start testing the deeper access requirements.

The controller service configuration provides multiple ways to access a bucket and a bunch of settings.

Make sure you have a working access/key credentials tested directly in the processor before moving to the Controller Service.