Connecting to AWS S3 from NiFi

[#1]

Hi,

i am trying to connect to an AWS S3 bucket which i was given access to.

I got uid,pwd and a key from the owners. But in ListS3 processor i dont see those fields.

i see Access Key and Secret Key properties. Is Access key and password are same ? We dont have to put in a user name.? also is Region a must.?

do i need to specify just the bucket name for ex vendor-company-bucket or do i have to fully specify the URL like https://vendor-company-bucket.s3-us-west-2.amazonaws.com?

Regards,

Sai

[#2 · Accepted solution]

@Saikrishna Tarapareddy

You need to provide your AWS Access Key and Secret Key, assuming that you are authorized to access the bucket. http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html

I have never used Nifi with S3 but my guess is that you wan to specify bucket name (not URL) and region like here https://community.hortonworks.com/articles/49467/integrating-apache-nifi-with-aws-s3-and-sqs.html, so what you did looks correct, just the AWS Access Key and Secret Key are missing.

[#3]

@Saikrishna Tarapareddy

You don't use username and pswd, you need access key and secret key. Admin of the S3 account can generate it (refer to https://aws.amazon.com/blogs/security/wheres-my-secret-access-key/)

Region and bucket are mandatory and you can see that when you access your S3 via web:

under "Services" menu go to "Storage->S3":

[#4]

I am facing different issue, I gave all the details you mentioned but i am its giving me an error unable to execute HTTP request. Note i used proxy also to connect but still unable to execute

@SumanP

[#5]

I am facing connectivity issue from Apache Nifi to Amazon SQS & S3:

I have given the details in the below link:

https://community.hortonworks.com/questions/214811/connection-issue-in-integrating-apache-nifi-with-...

Thanks in Advance.

Please suggest.

[#6]

Hi,
I m trying to put files into AWS S3 browser from Nifi using PutS3Object processor and getting below error.
Bucket Name ,  AccessKeyID and SecretAccessKey are used correctly.

I m stuck here. Please help.

[#7]

@Naveen_Sagar, Welcome to Cloudera Community! As this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post.

[#8]

 Sure Vidya

Thanks.!