Support Questions

Find answers, ask questions, and share your expertise

Looking for some help on how to use QueryElasticsearchHttp with self_signed certificate

avatar
Contributor

Hello NiFi team,

 

I'm new to NiFi, I'm trying to get data from Elasticsearch using QueryElasticsearchHttp however I have a self_signed certificate I'm not sure how to use that if you can give some examples or just some basic steps I can try to set that up on my end. 

Thank you in advance for the help.

 

2 ACCEPTED SOLUTIONS

avatar

@Vasu_  I am most definitely an expert in NIFI SSL Context Services and the various different ways to build the controller services with cacerts, public certs, and self signed certs.

 

Here is an article i just wrote about Modern NiFi and SSL:

https://community.cloudera.com/t5/Community-Articles/NIFI-SSL-in-Modern-Versions-of-NiFi/ta-p/371937

 

It is important to understand how to make a working SSL Context Service before trying to make custom ones.  So the examples in here to use nifi's own cert's keystore and truststore, and local java cacerts as keystore/trustore, will build confidence in how to configure the controller service before trying to build custom keystores and truststores yourself.

 

The following link has a technical example, and both solutions you can use here.  

  1. Use java cacerts if your elastic endpoint is a public signed ssl cert to build a SSL Context Service
  2. Build you own custom keystore and truststore using actual cert from the endpoint and use that to build a SSL Context Service

https://community.cloudera.com/t5/Support-Questions/Configure-StandardSSLContextService-for-Elastics...

 

View solution in original post

avatar

@Vasu_   Can you provide screen shots of config for the processor(s) and the SSL Context Service you created?

I can suggest more specific commands to build the keystore and truststore, but I will need to see what the hostname is for elastic.   Additionally, if you did attempt to create a keystore/truststore from the self signed cert,  be sure to share the commands you used.

 

Commands/Code would go in a Preformatted box (top right in the full wysiwig pannel) like this

View solution in original post

7 REPLIES 7

avatar
Community Manager

@Vasu_ Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our NiFi experts @cotopaul  who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.


Regards,

Diana Torres,
Community Moderator


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Learn more about the Cloudera Community:

avatar
Contributor

@cotopaul I have an elastic search instance trying to connect from my local Nifi and I get the below error

 

Failed to read from Elasticsearch due to PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target, this may indicate an error in configuration


I really appreciate your help.

 

avatar

yea, sorry, but this is a topic where I am no expert 😞 I do not really understand how certificates work and how they should be generated and used.

Nevertheless, if you are certain that your certificate is correct and you should be able to connect to your Elasticsearch, you should define it in your SSL Context Service and then proceed with configuring your NiFi processor in order to extract the data your need.

Here is how to configure the SSL Context Service for Elastic Search: https://community.cloudera.com/t5/Support-Questions/Configure-StandardSSLContextService-for-Elastics...

 

And here is an example on how you should configure the NiFi Processor:
https://nathanlabadie.com/streaming-from-elastic-to-syslog-via-apache-nifi/

avatar

@Vasu_  I am most definitely an expert in NIFI SSL Context Services and the various different ways to build the controller services with cacerts, public certs, and self signed certs.

 

Here is an article i just wrote about Modern NiFi and SSL:

https://community.cloudera.com/t5/Community-Articles/NIFI-SSL-in-Modern-Versions-of-NiFi/ta-p/371937

 

It is important to understand how to make a working SSL Context Service before trying to make custom ones.  So the examples in here to use nifi's own cert's keystore and truststore, and local java cacerts as keystore/trustore, will build confidence in how to configure the controller service before trying to build custom keystores and truststores yourself.

 

The following link has a technical example, and both solutions you can use here.  

  1. Use java cacerts if your elastic endpoint is a public signed ssl cert to build a SSL Context Service
  2. Build you own custom keystore and truststore using actual cert from the endpoint and use that to build a SSL Context Service

https://community.cloudera.com/t5/Support-Questions/Configure-StandardSSLContextService-for-Elastics...

 

avatar
Contributor

Hey @steven-matison Thank you so much for the articles. I tried all the possible options. Still no luck.

 

I have a "Self Signed" cert that I need to use to connect to the elastic search instance from NiFi please give me a step-by-step example as I'm totally new to NiFi, when I tested the same cert from Postman it's working as expected. Is there a dynamic parameter that I can use to point to the cert location from QueryElasticsearchHttp processor?

 

Here are the errors I get when I tried using the Java cacerts:

Failed to read from Elasticsearch due to PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target, this may indicate an error in configuration (hosts, username/password, etc.). Routing to retry: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

 

Thank you!!

avatar

@Vasu_   Can you provide screen shots of config for the processor(s) and the SSL Context Service you created?

I can suggest more specific commands to build the keystore and truststore, but I will need to see what the hostname is for elastic.   Additionally, if you did attempt to create a keystore/truststore from the self signed cert,  be sure to share the commands you used.

 

Commands/Code would go in a Preformatted box (top right in the full wysiwig pannel) like this

avatar
Contributor

Hey @steven-matison and @Edenwheeler thank you so much for your help It worked with StandardProxyConfigurationService controller services however I still have issues with StandardRestrictedSSLContextService controller service. Anyway, thank you so much for the help and details steps that helped me a lot.

 

Thank you!!