Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Connecting to AWS S3 from NiFi

avatar
Super Collaborator

Hi,

i am trying to connect to an AWS S3 bucket which i was given access to.

I got uid,pwd and a key from the owners. But in ListS3 processor i dont see those fields.

i see Access Key and Secret Key properties. Is Access key and password are same ? We dont have to put in a user name.? also is Region a must.?

do i need to specify just the bucket name for ex vendor-company-bucket or do i have to fully specify the URL like https://vendor-company-bucket.s3-us-west-2.amazonaws.com?

43829-lists3.jpg

Regards,

Sai

1 ACCEPTED SOLUTION

avatar

@Saikrishna Tarapareddy

You need to provide your AWS Access Key and Secret Key, assuming that you are authorized to access the bucket. http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html

I have never used Nifi with S3 but my guess is that you wan to specify bucket name (not URL) and region like here https://community.hortonworks.com/articles/49467/integrating-apache-nifi-with-aws-s3-and-sqs.html, so what you did looks correct, just the AWS Access Key and Secret Key are missing.

View solution in original post

4 REPLIES 4

avatar

@Saikrishna Tarapareddy

You need to provide your AWS Access Key and Secret Key, assuming that you are authorized to access the bucket. http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html

I have never used Nifi with S3 but my guess is that you wan to specify bucket name (not URL) and region like here https://community.hortonworks.com/articles/49467/integrating-apache-nifi-with-aws-s3-and-sqs.html, so what you did looks correct, just the AWS Access Key and Secret Key are missing.

avatar
Super Collaborator
@Saikrishna Tarapareddy

You don't use username and pswd, you need access key and secret key. Admin of the S3 account can generate it (refer to https://aws.amazon.com/blogs/security/wheres-my-secret-access-key/)

Region and bucket are mandatory and you can see that when you access your S3 via web:

42927-s3-1.png

under "Services" menu go to "Storage->S3":

42926-s3-2.png

avatar

I am facing different issue, I gave all the details you mentioned but i am its giving me an error unable to execute HTTP request. Note i used proxy also to connect but still unable to execute

@SumanP

avatar
Rising Star

I am facing connectivity issue from Apache Nifi to Amazon SQS & S3:

I have given the details in the below link:

https://community.hortonworks.com/questions/214811/connection-issue-in-integrating-apache-nifi-with-...

Thanks in Advance.

Please suggest.