Skip to main content
Techie123
Techie123
Silver ⭐️⭐️⭐️
February 9, 2023
Question

Getting Error While fetching files from s3 bucket using FetchS3Object Nifi

Hi Team,

 

I have an requirement to generate temporary credentials of AWS and fetch files from the S3 bucket  and use this credentials in FetchS3Object

But at FetchS3Object processor I am getting below error while fetching files from the bucket:

 

"FetchS3Object[id=xxx] Failed to retrieve S3 Object for FlowFile[filename=xyz.json]; routing to failure: com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain: [EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)), SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey), WebIdentityTokenCredentialsProvider: To use assume role profiles the aws-java-sdk-sts module must be on the class path., com.amazonaws.auth.profile.ProfileCredentialsProvider@3075e891: profile file cannot be null, com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper@2162bad1: Failed to connect to service endpoint: ]"   

 

can you please help me with the issue

3 replies

steven-matison
Platinum ⭐️⭐️
steven-matison
Platinum ⭐️⭐️
February 9, 2023

@Techie123 You are going to need to provide credentials for the nifi calls against any s3 bucket with access controls.   I would recommend working in lower nifi dev environments and use a public S3 buckets to get comfortable.   This will test basics of your flow without access issues.   This will also remove confusion around nifi flow functionality vs AWS access issues and help you learn when/where to use the different ways (key, or a controler service w/ credentials) to provide access from nifi to s3.   

Translate
Techie123
Techie123Author
Silver ⭐️⭐️⭐️
February 9, 2023

I need to use the controller service with credentials but still am getting the error. You have an Idea how can i do that. Any suggestions? 

Translate
steven-matison
Platinum ⭐️⭐️
steven-matison
Platinum ⭐️⭐️
February 9, 2023

@Techie123 Well, like i said,  you have to learn the aws side of providing access to a bucket.  A public bucket starting point will show you what you have to do, inside of the bucket config, to allow other systems to access that bucket.   For example starting from public open bucket, to whatever access control level you ultimately need to have.   Getting lost in that space is not necessarily a "nifi" thing....  so my recommendation is to build nifi with public bucket,  THEN when it works, start testing the deeper access requirements.

 

The controller service configuration provides multiple ways to access a bucket and a bunch of settings.

 

Screen Shot 2023-02-09 at 9.23.10 AM.png

 

  Make sure you have a working access/key credentials tested directly in the processor before moving to the Controller Service.

 

Screen Shot 2023-02-09 at 9.23.41 AM.png

 

Translate
    Terms & ConditionsAccessibility statement