Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Ambari checks for oozie failing - pki chain error

Ambari checks for oozie failing - pki chain error

Rising Star

Hello,

after i moved Oozie between server, ambari flags the server in Error as "Oozie admin -Oozie <URL> -status " returns a PKI check errors.

I have check all the key stores and trust store to be ok, the root cert is added to the Linux trusted CA (server was rebooted in case).

The same command from Linux Prompt works flawless.

Ambari-agent temp and cache have been cleared,

I could find where ambari-agent is retrieving his trust store.

Any ideas.?

Thanks

Christoohe

6 REPLIES 6

Re: Ambari checks for oozie failing - pki chain error

Rising Star

@Christophe Vico

Did you check the Oozie server logs for this issue ?

Further "Oozie admin -Oozie <URL> -status " error is on the oozie server Alert?

Can you post the complete alert Stack trace ?

Re: Ambari checks for oozie failing - pki chain error

Rising Star

@Swapan Shridhar

The Oozie logs are clean indeed so n regard with this issue (and running Oozie admin command from the Linux prompt does return a valid healthy status)

The PKI chain error is indeed in the Oozie alert (and matched in the ambari agent log file).

I'll post the stack tomorrow.

(I just saw I made a typo in original post : I could NOT locate what for truststore ambari agent relies on )

Thanks

Re: Ambari checks for oozie failing - pki chain error

Super Collaborator

@Christophe Vico

Ambari server has its own truststore. If you have SSL enabled on oozie webUI, make sure that you have the cert of it in ambari server truststore. You can check for truststore path set for ambari server in /etc/ambari-server/conf/ambari.properties. If not already done import the cert , you can create the cert file for the SSL enabled service using below command.

echo | openssl s_client -connect <OozieServerIP>:<Port> 2>&1 | sed --quiet '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > oozie.crt

Use keytool command to import the cert to ambari-server trust store:

#keytool -import -file oozie.crt -keystore <AmbariTrustStorePath> -alias oozie-cert -storepass <trustStorePassword>

Re: Ambari checks for oozie failing - pki chain error

Rising Star

@rguruvannagari

I checked and the truststore was already set as it should.

The problem seems to be located @the agent side, not on the server itself. See logs & report below.

I have re-checked the truststores on both sides, check the ownership & permissions to no luck. The ambari-agent does not list truststore in its config, so I'm not sure where to look here.

I'm still puzzled by the fact that oozie command works just fine when run form the prompt.

Thanks!

Ambari-agent log (on the nodes running Oozie server)


ERROR 2017-04-05 08:26:47,657 script_alert.py:119 - [Alert][oozie_server_status] Failed with result CRITICAL: ["Execution of 'source /usr/hdp/current/oozie-server/conf/oozie-env.sh ; oozie admin -oozie https://FQDNoozie:port/oozie -status' returned 255. Error: IO_ERROR : java.io.IOException: Error while connecting Oozie server. No of retries = 1. Exception = sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"]
ERROR 2017-04-05 08:26:47,657 script_alert.py:119 - [Alert][oozie_server_status] Failed with result CRITICAL: ["Execution of 'source /usr/hdp/current/oozie-server/conf/oozie-env.sh ; oozie admin -oozie https://FQDNoozie:port/oozie -status' returned 255. Error: IO_ERROR : java.io.IOException: Error while connecting Oozie server. No of retries = 1. Exception = sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"]
INFO 2017-04-05 08:26:59,681 Controller.py:277 - Heartbeat with server is running...


Oozie call from the command line
[user@server conf]$ oozie admin -oozie https://FQDNoozie:port/oozie -status
System mode: NORMAL








Ambari-alert stack


        Execution of 'source /usr/hdp/current/oozie-server/conf/oozie-env.sh ; oozie admin -oozie https://FQDNoozie:port/oozie -status' returned 255. Error: IO_ERROR : java.io.IOException: Error while connecting Oozie server. No of retries = 1. Exception = sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
      


Re: Ambari checks for oozie failing - pki chain error

Super Collaborator
@Christophe Vico

Ambari agent doest maintain any truststore, and the oozie_server_status alert is executed from Ambari server end and not from agent. So truststore in ambari server is expected to have the oozie server certificate. As you have moved oozie server to another host, the certificate it is holding might not have correct CN(in a certificate CN is expected to be the FQDN of the server running service). You may need to create new cert for oozie server after moving to different host and import that new cert to ambari-server truststore file.

Highlighted

Re: Ambari checks for oozie failing - pki chain error

Rising Star

@rguruvannagari

I did regenerate the cert, check the FQDN etc against the certificate used on the precedent machine: all seems good to me.

Still the same problem, ONLY for Ambari. I'm really puzzled :)

What I forgot to mention I realize: Oozie before was on the Ambari server.