Hello All, I have a 10 node HDP 2.6 (.1.0) cluster, I have installed Hortonworks SmartSense with HST Server on master and HST Agents on all the nodes (master and slave1-9). Its a fresh HDP install on RHEL 7. SmartSense installation went good and after the install I only had HST Agents running on 3 slaves (Slaves 1,2 & 3). Even though I restart all HST Agents, they still are live only on the same 3 slaves (Slaves 1,2 & 3). Can anyone please help me in troubleshooting this issue? Logs captured for the entire restart operation from /var/log/hst/hst-server.log INFO [main] CertificateManager:70 - Initialization of root certificate
INFO [main] CertificateManager:72 - Certificate exists:true
INFO [main] Configuration:562 - Reading password from existing file
WARN [main] ConfigChangeListener:155 - Creating a patch
INFO [main] ConfigChangeListener:236 - Patch created : /var/lib/smartsense/hst-server/updates/upload/config-update.tgz
INFO [main] SupportToolServer:572 - Bundle Purge Scheduler enabled at :Thu Oct 19 13:30:21 EDT 2017. Bundle Purge job will run every 24 hrs.
INFO [main] Server:266 - jetty-7.6.7.v20120910
INFO [main] ContextHandler:744 - started o.e.j.s.ServletContextHandler{/,file:/usr/hdp/share/hst/hst-server/web/}
INFO [main] AbstractConnector:338 - Started SelectChannelConnector@0.0.0.0:9000
INFO [main] Server:266 - jetty-7.6.7.v20120910
INFO [main] ContextHandler:744 - started o.e.j.s.ServletContextHandler{/,null}
INFO [main] SslContextFactory:300 - Enabled Protocols [SSLv2Hello, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
INFO [main] AbstractConnector:338 - Started SslSelectChannelConnector@0.0.0.0:9440
INFO [main] SslContextFactory:300 - Enabled Protocols [SSLv2Hello, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
INFO [main] AbstractConnector:338 - Started SslSelectChannelConnector@0.0.0.0:9441
WARN [qtp943219925-85] nio:651 - javax.net.ssl.SSLException: Received fatal alert: unknown_ca
WARN [qtp943219925-86] nio:651 - javax.net.ssl.SSLException: Received fatal alert: unknown_ca
INFO [qtp943219925-85] SupportToolResource:142 - Unregistering agent slave3
INFO [qtp943219925-86] SupportToolResource:142 - Unregistering agent slave2
WARN [qtp943219925-87] nio:651 - javax.net.ssl.SSLException: Received fatal alert: unknown_ca
WARN [qtp943219925-87] nio:651 - javax.net.ssl.SSLException: Received fatal alert: unknown_ca
INFO [qtp943219925-85] CertificateManager:189 - Signing of agent certificate
INFO [qtp943219925-85] CertificateManager:190 - Verifying passphrase
INFO [qtp943219925-85] Configuration:562 - Reading password from existing file
INFO [qtp943219925-85] CertificateManager:214 - Revoking of slave3 certificate.
WARN [qtp943219925-87] nio:651 - javax.net.ssl.SSLException: Received fatal alert: unknown_ca
INFO [qtp943219925-85] CertificateManager:265 - Command openssl ca -config /var/lib/smartsense/hst-server/keys/ca.config -keyfile /var/lib/smartsense/hst-server/keys/ca.key -revoke /var/lib/smartsense/hst-server/keys/slave3.crt -batch -passin pass:**** -cert /var/lib/smartsense/hst-server/keys/ca.crt was finished with exit code: 0 - the operation was completely successfully.
WARN [qtp943219925-87] nio:651 - javax.net.ssl.SSLException: Received fatal alert: unknown_ca
WARN [qtp943219925-88] nio:651 - javax.net.ssl.SSLException: Received fatal alert: unknown_ca
INFO [qtp943219925-85] CertificateManager:265 - Command openssl ca -config /var/lib/smartsense/hst-server/keys/ca.config -in /var/lib/smartsense/hst-server/keys/slave3.csr -out /var/lib/smartsense/hst-server/keys/slave3.crt -batch -md sha256 -passin pass:**** -keyfile /var/lib/smartsense/hst-server/keys/ca.key -cert /var/lib/smartsense/hst-server/keys/ca.crt was finished with exit code: 0 - the operation was completely successfully.
INFO [qtp943219925-89] SupportToolResource:142 - Unregistering agent slave1
WARN [qtp943219925-87] nio:651 - javax.net.ssl.SSLException: Received fatal alert: unknown_ca
WARN [qtp943219925-90] nio:651 - javax.net.ssl.SSLException: Received fatal alert: unknown_ca
WARN [qtp943219925-85] nio:651 - javax.net.ssl.SSLException: Received fatal alert: unknown_ca
INFO [qtp943219925-90] SupportToolResource:115 - Registering agent, id=slave3, version=1.4.0.2.5.0.3-7
WARN [qtp943219925-90] nio:651 - javax.net.ssl.SSLException: Received fatal alert: unknown_ca
WARN [qtp943219925-88] nio:651 - javax.net.ssl.SSLException: Received fatal alert: unknown_ca
WARN [qtp943219925-90] nio:651 - javax.net.ssl.SSLException: Received fatal alert: unknown_ca
WARN [qtp943219925-86] nio:651 - javax.net.ssl.SSLException: Received fatal alert: unknown_ca
INFO [Thread-3] FileWatcher:232 - Watcher configuration has been changed. re-initializing watcher.
INFO [Thread-3] ConfigChangeListener:131 - listner configuration has been changed. re-initializing listner.
WARN [qtp943219925-87] nio:651 - javax.net.ssl.SSLException: Received fatal alert: unknown_ca
WARN [qtp943219925-87] nio:651 - javax.net.ssl.SSLException: Received fatal alert: unknown_ca
WARN [qtp943219925-87] nio:651 - javax.net.ssl.SSLException: Received fatal alert: unknown_ca
WARN [qtp943219925-88] nio:651 - javax.net.ssl.SSLException: Received fatal alert: unknown_ca
... View more
