Skip to main content
Vasu_
Vasu_
Silver ⭐️⭐️⭐️
June 7, 2023
Solved

Looking for some help on how to use QueryElasticsearchHttp with self_signed certificate

Hello NiFi team,

 

I'm new to NiFi, I'm trying to get data from Elasticsearch using QueryElasticsearchHttp however I have a self_signed certificate I'm not sure how to use that if you can give some examples or just some basic steps I can try to set that up on my end. 

Thank you in advance for the help.

 

steven-matison
steven-matisonAnswer
Platinum ⭐️⭐️
June 8, 2023

@Vasu_  I am most definitely an expert in NIFI SSL Context Services and the various different ways to build the controller services with cacerts, public certs, and self signed certs.

 

Here is an article i just wrote about Modern NiFi and SSL:

https://community.cloudera.com/t5/Community-Articles/NIFI-SSL-in-Modern-Versions-of-NiFi/ta-p/371937

 

It is important to understand how to make a working SSL Context Service before trying to make custom ones.  So the examples in here to use nifi's own cert's keystore and truststore, and local java cacerts as keystore/trustore, will build confidence in how to configure the controller service before trying to build custom keystores and truststores yourself.

 

The following link has a technical example, and both solutions you can use here.  

  1. Use java cacerts if your elastic endpoint is a public signed ssl cert to build a SSL Context Service
  2. Build you own custom keystore and truststore using actual cert from the endpoint and use that to build a SSL Context Service

https://community.cloudera.com/t5/Support-Questions/Configure-StandardSSLContextService-for-Elasticsearch/m-p/302719

 

View in thread

7 replies

DianaTorres
Community Manager
DianaTorres
Community Manager
June 7, 2023

@Vasu_ Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our NiFi experts @cotopaul  who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

Translate
Regards, Diana Torres, Senior Community Moderator
Vasu_
Vasu_Author
Silver ⭐️⭐️⭐️
June 7, 2023

@cotopaul I have an elastic search instance trying to connect from my local Nifi and I get the below error

 

Failed to read from Elasticsearch due to PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target, this may indicate an error in configuration


I really appreciate your help.

 

Translate
cotopaul
cotopaul
Platinum ⭐️
June 8, 2023

yea, sorry, but this is a topic where I am no expert 😞 I do not really understand how certificates work and how they should be generated and used.

Nevertheless, if you are certain that your certificate is correct and you should be able to connect to your Elasticsearch, you should define it in your SSL Context Service and then proceed with configuring your NiFi processor in order to extract the data your need.

Here is how to configure the SSL Context Service for Elastic Search: https://community.cloudera.com/t5/Support-Questions/Configure-StandardSSLContextService-for-Elasticsearch/m-p/302719

 

And here is an example on how you should configure the NiFi Processor:
https://nathanlabadie.com/streaming-from-elastic-to-syslog-via-apache-nifi/

Translate
steven-matison
Platinum ⭐️⭐️
steven-matisonAnswer
Platinum ⭐️⭐️
June 8, 2023

@Vasu_  I am most definitely an expert in NIFI SSL Context Services and the various different ways to build the controller services with cacerts, public certs, and self signed certs.

 

Here is an article i just wrote about Modern NiFi and SSL:

https://community.cloudera.com/t5/Community-Articles/NIFI-SSL-in-Modern-Versions-of-NiFi/ta-p/371937

 

It is important to understand how to make a working SSL Context Service before trying to make custom ones.  So the examples in here to use nifi's own cert's keystore and truststore, and local java cacerts as keystore/trustore, will build confidence in how to configure the controller service before trying to build custom keystores and truststores yourself.

 

The following link has a technical example, and both solutions you can use here.  

  1. Use java cacerts if your elastic endpoint is a public signed ssl cert to build a SSL Context Service
  2. Build you own custom keystore and truststore using actual cert from the endpoint and use that to build a SSL Context Service

https://community.cloudera.com/t5/Support-Questions/Configure-StandardSSLContextService-for-Elasticsearch/m-p/302719

 

Translate
    Vasu_
    Vasu_Author
    Silver ⭐️⭐️⭐️
    June 8, 2023

    Hey @steven-matison Thank you so much for the articles. I tried all the possible options. Still no luck.

     

    I have a "Self Signed" cert that I need to use to connect to the elastic search instance from NiFi please give me a step-by-step example as I'm totally new to NiFi, when I tested the same cert from Postman it's working as expected. Is there a dynamic parameter that I can use to point to the cert location from QueryElasticsearchHttp processor?

     

    Here are the errors I get when I tried using the Java cacerts:

    Failed to read from Elasticsearch due to PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target, this may indicate an error in configuration (hosts, username/password, etc.). Routing to retry: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

     

    Thank you!!

    Translate
    steven-matison
    Platinum ⭐️⭐️
    steven-matison
    Platinum ⭐️⭐️
    June 9, 2023

    @Vasu_   Can you provide screen shots of config for the processor(s) and the SSL Context Service you created?

    I can suggest more specific commands to build the keystore and truststore, but I will need to see what the hostname is for elastic.   Additionally, if you did attempt to create a keystore/truststore from the self signed cert,  be sure to share the commands you used.

     

    Commands/Code would go in a Preformatted box (top right in the full wysiwig pannel) like this

    Translate
    Vasu_
    Vasu_Author
    Silver ⭐️⭐️⭐️
    June 9, 2023

    Hey @steven-matison and @Former Member thank you so much for your help It worked with StandardProxyConfigurationService controller services however I still have issues with StandardRestrictedSSLContextService controller service. Anyway, thank you so much for the help and details steps that helped me a lot.

     

    Thank you!!

     

    Translate
      Terms & ConditionsAccessibility statement