question Re: Connecting to AWS S3 from NiFi in Archives of Support Questions (Read Only)

Connecting to AWS S3 from NiFi

saikrishna_tara — Sun, 18 Aug 2019 02:46:16 GMT

Hi,

i am trying to connect to an AWS S3 bucket which i was given access to.

I got uid,pwd and a key from the owners. But in ListS3 processor i dont see those fields.

i see Access Key and Secret Key properties. Is Access key and password are same ? We dont have to put in a user name.? also is Region a must.?

do i need to specify just the bucket name for ex vendor-company-bucket or do i have to fully specify the URL like https://vendor-company-bucket.s3-us-west-2.amazonaws.com?

Regards,

Sai

Re: Connecting to AWS S3 from NiFi

Dominika — Wed, 06 Dec 2017 02:29:05 GMT

@Saikrishna Tarapareddy

You need to provide your AWS Access Key and Secret Key, assuming that you are authorized to access the bucket. http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html

I have never used Nifi with S3 but my guess is that you wan to specify bucket name (not URL) and region like here https://community.hortonworks.com/articles/49467/integrating-apache-nifi-with-aws-s3-and-sqs.html, so what you did looks correct, just the AWS Access Key and Secret Key are missing.

Re: Connecting to AWS S3 from NiFi

eberezitsky — Sun, 18 Aug 2019 02:46:08 GMT

@Saikrishna Tarapareddy

You don't use username and pswd, you need access key and secret key. Admin of the S3 account can generate it (refer to https://aws.amazon.com/blogs/security/wheres-my-secret-access-key/)

Region and bucket are mandatory and you can see that when you access your S3 via web:

under "Services" menu go to "Storage->S3":

Re: Connecting to AWS S3 from NiFi

suman_prabhala — Fri, 02 Feb 2018 01:40:55 GMT

I am facing different issue, I gave all the details you mentioned but i am its giving me an error unable to execute HTTP request. Note i used proxy also to connect but still unable to execute

@SumanP

Re: Connecting to AWS S3 from NiFi

ClouderaVMissue — Sat, 25 Aug 2018 16:23:15 GMT

I am facing connectivity issue from Apache Nifi to Amazon SQS & S3:

I have given the details in the below link:

https://community.hortonworks.com/questions/214811/connection-issue-in-integrating-apache-nifi-with-a.html

Thanks in Advance.

Please suggest.

Re: Connecting to AWS S3 from NiFi

Naveen_Sagar — Wed, 29 May 2024 09:23:28 GMT

Hi,
I m trying to put files into AWS S3 browser from Nifi using PutS3Object processor and getting below error.
Bucket Name , AccessKeyID and SecretAccessKey are used correctly.

I m stuck here. Please help.

Re: Connecting to AWS S3 from NiFi

VidyaSargur — Wed, 29 May 2024 13:49:06 GMT

@Naveen_Sagar, Welcome to Cloudera Community! As this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post.

Re: Connecting to AWS S3 from NiFi

Naveen_Sagar — Fri, 31 May 2024 06:48:59 GMT

Sure Vidya

Thanks.!