Looking for some help on how to use QueryElasticsearchHttp with self_signed certificate

[#1]

Hello NiFi team,

I'm new to NiFi, I'm trying to get data from Elasticsearch using QueryElasticsearchHttp however I have a self_signed certificate I'm not sure how to use that if you can give some examples or just some basic steps I can try to set that up on my end. 

Thank you in advance for the help.

[#2 · Accepted solution]

@Vasu_  I am most definitely an expert in NIFI SSL Context Services and the various different ways to build the controller services with cacerts, public certs, and self signed certs.

Here is an article i just wrote about Modern NiFi and SSL:

https://community.cloudera.com/t5/Community-Articles/NIFI-SSL-in-Modern-Versions-of-NiFi/ta-p/371937

It is important to understand how to make a working SSL Context Service before trying to make custom ones.  So the examples in here to use nifi's own cert's keystore and truststore, and local java cacerts as keystore/trustore, will build confidence in how to configure the controller service before trying to build custom keystores and truststores yourself.

The following link has a technical example, and both solutions you can use here.  

1. Use java cacerts if your elastic endpoint is a public signed ssl cert to build a SSL Context Service
2. Build you own custom keystore and truststore using actual cert from the endpoint and use that to build a SSL Context Service

https://community.cloudera.com/t5/Support-Questions/Configure-StandardSSLContextService-for-Elastics...

[#3]

@Vasu_ Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our NiFi experts @cotopaul  who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

[#4]

@cotopaul I have an elastic search instance trying to connect from my local Nifi and I get the below error

Failed to read from Elasticsearch due to PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target, this may indicate an error in configuration

I really appreciate your help.

[#5]

yea, sorry, but this is a topic where I am no expert 😞 I do not really understand how certificates work and how they should be generated and used.

Nevertheless, if you are certain that your certificate is correct and you should be able to connect to your Elasticsearch, you should define it in your SSL Context Service and then proceed with configuring your NiFi processor in order to extract the data your need.

Here is how to configure the SSL Context Service for Elastic Search: https://community.cloudera.com/t5/Support-Questions/Configure-StandardSSLContextService-for-Elastics...

And here is an example on how you should configure the NiFi Processor:
https://nathanlabadie.com/streaming-from-elastic-to-syslog-via-apache-nifi/

[#6 · Accepted solution]

@Vasu_  I am most definitely an expert in NIFI SSL Context Services and the various different ways to build the controller services with cacerts, public certs, and self signed certs.

Here is an article i just wrote about Modern NiFi and SSL:

https://community.cloudera.com/t5/Community-Articles/NIFI-SSL-in-Modern-Versions-of-NiFi/ta-p/371937

It is important to understand how to make a working SSL Context Service before trying to make custom ones.  So the examples in here to use nifi's own cert's keystore and truststore, and local java cacerts as keystore/trustore, will build confidence in how to configure the controller service before trying to build custom keystores and truststores yourself.

The following link has a technical example, and both solutions you can use here.  

1. Use java cacerts if your elastic endpoint is a public signed ssl cert to build a SSL Context Service
2. Build you own custom keystore and truststore using actual cert from the endpoint and use that to build a SSL Context Service

https://community.cloudera.com/t5/Support-Questions/Configure-StandardSSLContextService-for-Elastics...

[#7]

Hey @steven-matison Thank you so much for the articles. I tried all the possible options. Still no luck.

I have a "Self Signed" cert that I need to use to connect to the elastic search instance from NiFi please give me a step-by-step example as I'm totally new to NiFi, when I tested the same cert from Postman it's working as expected. Is there a dynamic parameter that I can use to point to the cert location from QueryElasticsearchHttp processor?

Here are the errors I get when I tried using the Java cacerts:

Failed to read from Elasticsearch due to PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target, this may indicate an error in configuration (hosts, username/password, etc.). Routing to retry: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Thank you!!

[#8 · Accepted solution]

@Vasu_   Can you provide screen shots of config for the processor(s) and the SSL Context Service you created?

I can suggest more specific commands to build the keystore and truststore, but I will need to see what the hostname is for elastic.   Additionally, if you did attempt to create a keystore/truststore from the self signed cert,  be sure to share the commands you used.

Commands/Code would go in a Preformatted box (top right in the full wysiwig pannel) like this

[#9]

Hey @steven-matison and @Edenwheeler thank you so much for your help It worked with StandardProxyConfigurationService controller services however I still have issues with StandardRestrictedSSLContextService controller service. Anyway, thank you so much for the help and details steps that helped me a lot.

Thank you!!