Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Error while using fetchs3object nifi processor to fetch data from aws s3

Error while using fetchs3object nifi processor to fetch data from aws s3

New Contributor

fetchs3.PNG

I am using Nifi Version 1.5.0.3.1.2.0-7.I am trying to fetch data from aws s3 storage by using fetchs3object processor.I had attached the screenshot of processor.I am trying to connect through aws access key and security key which i had already provided in the configuration details of fetchs3object prcessor.I had also provided bucket name,object key,region as shown in image.I am trying to connect to aws from nifi first time.It is showing error 'unable to find valid certification path to requested target' while starting the processor.

Please help me in resolving the error.

7 REPLIES 7
Highlighted

Re: Error while using fetchs3object nifi processor to fetch data from aws s3

Master Collaborator

@mayank_tripathi The solution here is to setup the SSL context Service for the processor.  To do this you will need to create a new SSL Context Service and provide it a path to keystore and/or truststore files which contain the s3 bucket’s SSL CERT.  The files should be on all nifi nodes, owned by nifi user.

 

 

 


 


If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  


 


Thanks,



Steven

Highlighted

Re: Error while using fetchs3object nifi processor to fetch data from aws s3

New Contributor

Thanks for replying.

Please help me regarding these points are:

1)Does we need keystore and truststore file both or using any of them would work.

2)How we can create keystore and truststore files.

Highlighted

Re: Error while using fetchs3object nifi processor to fetch data from aws s3

Master Collaborator

For your example I would use truststore to "Trust Amazon's Cert".  You get the Amazon Cert, and create truststore with it.   There are many ways to get the Amazon Cert.  Recently, I found that a very easy way (windows) is to use a browser.  Visit an https url to your s3 host.  Click the SSL link, view certificate, download, etc.   On Nifi node add this cert file to /etc/nifi/ssl/ as amazon.cer.  Next the truststore command looks like:

 

/usr/jdk64/jdk1.8.0_112/bin/keytool -import -file /etc/nifi/ssl/amazon.cer -alias amazon -keystore /etc/nifi/ssl/truststore-amazon.jks

 

Make sure the file is chown nifi:nifi (owned to nifi user) and copied to all nodes.

 

The SSLContext controller service is then configured as:

 

Screen Shot 2020-04-14 at 12.27.35 PM.png

 


 


If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  


 


Thanks,



Steven

Highlighted

Re: Error while using fetchs3object nifi processor to fetch data from aws s3

New Contributor

Thanks for replying.

1)I didn't understand exactly the way you suggested to download ssl certification.Please clear the steps once .

2)Please confirm that this is a free service or paid service.

Highlighted

Re: Error while using fetchs3object nifi processor to fetch data from aws s3

Master Collaborator

There are many ways to get a public cert.  I just recommend one I used recently.  You basically just click the lock in your browser.  In windows you can then download files from there.    If you are not on windows, or using command line you can reference this link for some other ways to get the public cert for amazon s3 urls:

 

https://serverfault.com/questions/661978/displaying-a-remote-ssl-certificate-details-using-cli-tools

 

Not sure what your last question refers to?

 


 


If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  


 


Thanks,



Steven

Highlighted

Re: Error while using fetchs3object nifi processor to fetch data from aws s3

New Contributor

1.PNG

Thanks for replying.

I had attached the screenshot of .cer file i had downloaded.

1)Please confirm the below steps

Now i will add .cer file at /etc/nifi/ssl/ as amazon.cer and trustore file at /trial as truststore-amazon.jks.I will change the jdk version as per version present on my cluster in truststore file.The content of truststore file will be like

/usr/jdk64/jdk1.8.0_77/bin/keytool -import -file /etc/nifi/ssl/amazon.cer -alias amazon -keystore /etc/nifi/ssl/truststore-amazon.jks

I will mention the truststore filename as /trial/truststore-amazon.jks in the configuration of fecths3 processor.

2)Using ssl certification is free service or i need to pay to amazon for it.

Highlighted

Re: Error while using fetchs3object nifi processor to fetch data from aws s3

Master Collaborator

@mayank_tripathi  It's free.  No issues there.

 

 

Yes I believe your summary is correct.  If you are using /trial folder, you can put the cer in there, and then execute the trustore command.  I like to keep all my files in the same place.   Just make sure when you are done that /trial folder is right permissions so nifi user can read the files and the files are copied to all nifi nodes.    If you do not do correct ownership and copy to all nodes part, the controller service will throw an error.  

 


 


If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  


 


Thanks,



Steven

Don't have an account?
Coming from Hortonworks? Activate your account here