Created 06-06-2023 05:24 PM
Hello NiFi team,
I'm new to NiFi, I'm trying to get data from Elasticsearch using QueryElasticsearchHttp however I have a self_signed certificate I'm not sure how to use that if you can give some examples or just some basic steps I can try to set that up on my end.
Thank you in advance for the help.
Created on 06-08-2023 06:24 AM - edited 06-08-2023 06:27 AM
@Vasu_ I am most definitely an expert in NIFI SSL Context Services and the various different ways to build the controller services with cacerts, public certs, and self signed certs.
Here is an article i just wrote about Modern NiFi and SSL:
https://community.cloudera.com/t5/Community-Articles/NIFI-SSL-in-Modern-Versions-of-NiFi/ta-p/371937
It is important to understand how to make a working SSL Context Service before trying to make custom ones. So the examples in here to use nifi's own cert's keystore and truststore, and local java cacerts as keystore/trustore, will build confidence in how to configure the controller service before trying to build custom keystores and truststores yourself.
The following link has a technical example, and both solutions you can use here.
Created on 06-09-2023 04:05 AM - edited 06-09-2023 04:05 AM
@Vasu_ Can you provide screen shots of config for the processor(s) and the SSL Context Service you created?
I can suggest more specific commands to build the keystore and truststore, but I will need to see what the hostname is for elastic. Additionally, if you did attempt to create a keystore/truststore from the self signed cert, be sure to share the commands you used.
Commands/Code would go in a Preformatted box (top right in the full wysiwig pannel) like this
Created 06-06-2023 10:53 PM
@Vasu_ Welcome to the Cloudera Community!
To help you get the best possible solution, I have tagged our NiFi experts @cotopaul who may be able to assist you further.
Please keep us updated on your post, and we hope you find a satisfactory solution to your query.
Regards,
Diana Torres,Created 06-07-2023 04:15 PM
@cotopaul I have an elastic search instance trying to connect from my local Nifi and I get the below error
Failed to read from Elasticsearch due to PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target, this may indicate an error in configuration
I really appreciate your help.
Created 06-08-2023 12:24 AM
yea, sorry, but this is a topic where I am no expert 😞 I do not really understand how certificates work and how they should be generated and used.
Nevertheless, if you are certain that your certificate is correct and you should be able to connect to your Elasticsearch, you should define it in your SSL Context Service and then proceed with configuring your NiFi processor in order to extract the data your need.
Here is how to configure the SSL Context Service for Elastic Search: https://community.cloudera.com/t5/Support-Questions/Configure-StandardSSLContextService-for-Elastics...
And here is an example on how you should configure the NiFi Processor:
https://nathanlabadie.com/streaming-from-elastic-to-syslog-via-apache-nifi/
Created on 06-08-2023 06:24 AM - edited 06-08-2023 06:27 AM
@Vasu_ I am most definitely an expert in NIFI SSL Context Services and the various different ways to build the controller services with cacerts, public certs, and self signed certs.
Here is an article i just wrote about Modern NiFi and SSL:
https://community.cloudera.com/t5/Community-Articles/NIFI-SSL-in-Modern-Versions-of-NiFi/ta-p/371937
It is important to understand how to make a working SSL Context Service before trying to make custom ones. So the examples in here to use nifi's own cert's keystore and truststore, and local java cacerts as keystore/trustore, will build confidence in how to configure the controller service before trying to build custom keystores and truststores yourself.
The following link has a technical example, and both solutions you can use here.
Created 06-08-2023 02:35 PM
Hey @steven-matison Thank you so much for the articles. I tried all the possible options. Still no luck.
I have a "Self Signed" cert that I need to use to connect to the elastic search instance from NiFi please give me a step-by-step example as I'm totally new to NiFi, when I tested the same cert from Postman it's working as expected. Is there a dynamic parameter that I can use to point to the cert location from QueryElasticsearchHttp processor?
Here are the errors I get when I tried using the Java cacerts:
Thank you!!
Created on 06-09-2023 04:05 AM - edited 06-09-2023 04:05 AM
@Vasu_ Can you provide screen shots of config for the processor(s) and the SSL Context Service you created?
I can suggest more specific commands to build the keystore and truststore, but I will need to see what the hostname is for elastic. Additionally, if you did attempt to create a keystore/truststore from the self signed cert, be sure to share the commands you used.
Commands/Code would go in a Preformatted box (top right in the full wysiwig pannel) like this
Created on 06-09-2023 04:11 PM - edited 06-09-2023 04:16 PM
Hey @steven-matison and @Edenwheeler thank you so much for your help It worked with StandardProxyConfigurationService controller services however I still have issues with StandardRestrictedSSLContextService controller service. Anyway, thank you so much for the help and details steps that helped me a lot.
Thank you!!