Hello, I am running into the same issue. We use Centos 7.5 version on our HDP cluster nodes and RHEL updates ran on the ambari-server few days ago, and now I could not access our ambari-server's URL in the browser, as the connection gets dropped. And I see the following in the Ambari-server's logs: javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate) And when I run curl to the Ambari-Server's HTTPS url: * Initializing NSS with certpath: sql:/etc/pki/nssdb * CAfile: /etc/pki/tls/certs/ca-bundle.crt CApath: none * NSS error -5938 (PR_END_OF_FILE_ERROR) * Encountered end of file * Closing connection 0 My current environment: Ambari Server 2.4.0 for HDP 2.5.0. Python version 2.7.5-69.el7_5 and Openssl version was updated in this server update, to 1.0.2k-12.el7 Should I downgrade to a different build of Python 2.7.5? I always had Python Cert verification disabled, that did not help. Other Options tried: a) Updated the /etc/ambari-server/conf/ambari.properties with the following line: security.server.disabled.protocols=SSL|SSLv2|SSLv2Hello|SSLv3|TLSv1 b) Updated the /etc/ambari-agent/conf/ambari-agent.ini with the following line: force_https_protocol=PROTOCOL_TLSv1_2 Restarted the Ambari-server and the agent after the above update to configuration files, still not working. Kindly please advise. ... View more

So I did a few config changes and restarted mysql couple of times, but not sure which one fixed it. Started with Ambari 2.7.0.0 installation on a fresh Centos 7.5 vm Installed the MariaDB 10.2 packages as described in the install document and in addition installed the MariaDB-devel as well. Increased the following as per the above Ambari Server Tuning documentation link: 'agent.threadpool.size.max' in /etc/ambari-server/conf/ambari.properties to 50 Increased the 'client.threadpool.size.max' in /etc/ambari-server/conf/ambari.properties to 100 Installed the mysql-connector-java version 8.0.12 using the commands given above (same available in Ambari documentation) and ran 'ambari server-setup' with the required options and started Ambari Server. At this point, I was still getting the C3P0 deadlock errors. Then I restarted mysql and ambari-server few times (didnot make any config updates for mysql though; and no changes to any of the Ambari Connection Pool tuning parameters); at one point, I noticed an error related to timezone 'EDT' not being recognized, so to fix that, I executed the following in MariaDB: set GLOBAL time_zone ='-5:00' And then restarted Ambari server again, and it started up. So I am not sure which of the above fixed this. Once I get HDP 3.0 installed successfully, I will probably retry the whole installation to narrow down the fix. Thank you very much for all of your inputs. ... View more

Okay, I will try this option as well, once I finish up with rest of the installation. Currently we have hdp 2.5 running which is configured with mariadb, hence I tried that first. Thank you. ... View more

Not yet, same C3P0 errors. I will redo a fresh installation again. Will keep posted. ... View more

Hi @Akhil S Naik, 
 I have restarted Ambari server plenty of times. There is no 'Caused by' section in the stack trace. Adding to the above error logs, I see 3 PoolThreadStacktrace for 3 helper threads and followed by a timeout message. Please find below: Thread[C3P0PooledConnectionPoolManager[identityToken->1bqv1ac9x130woy214hz5t|66982506]-HelperThread-#0,5,main]
java.net.PlainSocketImpl.socketConnect(Native Method)
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
java.net.Socket.connect(Socket.java:589)
java.net.Socket.connect(Socket.java:538)
java.net.Socket.<init>(Socket.java:434)
java.net.Socket.<init>(Socket.java:244)
com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:259)
com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:307)
com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2484)
com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2521)
com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2306)
com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:839)
com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:49)
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
java.lang.reflect.Constructor.newInstance(Constructor.java:423)
com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:421)
com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:350)
com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:175)
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:220)
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:206)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:203)
com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1138)
com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1125)
com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:44)
com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1870)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696)

WARN [C3P0PooledConnectionPoolManager[identityToken->1bqv1ac9x130woy214hz5t|66982506]-AdminTaskTimer] ThreadPoolAsynchronousRunner:220 - Task com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@5d1f3439 (in deadlocked PoolThread) failed to complete in maximum time 60000ms. Trying interrupt().
WARN [C3P0PooledConnectionPoolManager[identityToken->1bqv1ac9x130woy214hz5t|66982506]-AdminTaskTimer] ThreadPoolAsynchronousRunner:220 - Task com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@1204c5e (in deadlocked PoolThread) failed to complete in maximum time 60000ms. Trying interrupt().
[C3P0PooledConnectionPoolManager[identityToken->1bqv1ac9x130woy214hz5t|66982506]-AdminTaskTimer] ThreadPoolAsynchronousRunner:220 - Task com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@38cf7e05 (in deadlocked PoolThread) failed to complete in maximum time 60000ms. Trying interrupt().
2018-08-30 17:08:56,973 WARN [C3P0PooledConnectionPoolManager[identityToken->1bqv1ac9x130woy214hz5t|66982506]-AdminTaskTimer] ThreadPoolAsynchronousRunner:220 - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@f3bced7 -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks ... View more

Hi @sjanardhan, I will review the ambari-server tuning article for the ThreadPool related settings. Thanks for the link. This is a fresh installation of Ambari 2.7.0.0, but I noticed I am using a Centos7.5 image; Per the support matrix, it is supported until centos7.4 only; not sure if the above issue of C3P0 Deadlocks might be caused due to the Centos Versions.. ... View more

I have Zeppelin 0.7 configured as a separate service outside of Ambari, but livy sessions are not getting created and crashing. Could someone share the livy interpreter configuration to get this working in Zeppelin 0.7? Thanks in advance! ... View more

Thanks for the instructions. I have a HDP 2.5 cluster and want to move or create all the collection configuration to HDFS directory, instead of local disk. The config you have above is to update the solrconfig.xml for each collection and this works, but is there a way to update the entire thing from Ambari Console by updating the infra-solr-env-template? Thanks in advance for your input. ... View more

This worked. Thanks for the detailed steps. ... View more

Follow this article https://community.hortonworks.com/articles/88259/set-time-to-live-ttl-on-solr-records.html to set the values for the fields _ttl_ and _expire_at_ fields in the solrconfig.xml and related files and upload them to zookeeper and restart ambari-infra-solr services. That will help reducing the disk space usage according to the value of TTL setting. ... View more

Thanks for your article on setting TTL for Solr documents. However, in my environment, I have Ambari Infra-solr auto created cores for hadoop logs that are taking up disk space. I followed the above and updated the managed-schema and solrconfig.xml under /usr/lib/ambari-infra-solr/server/solr/configsets/data_driven_schema_configs/ I used Ambari Dashboard to restart Ambari Infra Solr and Zookeeper services instead of manually starting Solr using your above command. How would we know if Zookeeper and Solr picked up these settings. Thanks Anitha ... View more

I am running into errors from Ambari-infra-solr in HDP 2.5 with a Kerberized and SSL enabled cluster. I noticed that your steps have a separate keytab for solr-spnego. Is this mandatory to do this way? SOLR_KERB_KEYTAB=/etc/security/keytabs/solr-spnego.service.keytab The errors I have are: SASL configuration failed: javax.security.auth.login.LoginException: Pre-authentication information was invalid (24) Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it and '401 Authentication required' Please let me know what I am missing here. ... View more

No, I have been having lot of issues with Ambari Infra Solr, Kerberos, and HTTPS Logsearch UI. Still researching... ... View more

Thanks for the instructions. I have a Kerberos and HTTPS HDP 2.5 cluster, and after Kerberizing, I see errors in the Ambari infra logs of both the nodes that it could not replicate the index between the solr nodes. Is this related to the above steps? **Note: I am able to access the Solr UI of both the Ambari infra solr nodes, though. Errors: ERROR [c:hadoop_logs s:shard3 r:core_node1 x:hadoop_logs_shard3_replica1] org.apache.solr.update.StreamingSolrClients$1 (StreamingSolrClients.java:79) - error org.apache.solr.common.SolrException: Authentication required and when I restart the Ambari-Infra solr services, I do see the below error as well: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/> <title>Error 401 Authentication required</title> </head> <body><h2>HTTP ERROR 401</h2> <p>Problem accessing /solr/vertex_index_shard1_replica1/get. Reason: <pre> Authentication required</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/> </body> </html> Any inputs on how to resolve this? Recently, I updated the LogSearch UI to HTTPS and that is unable to connect to the Solr instances .. Thanks in advance. ... View more

Any one else faced the same issue with WebHcat trying to write to Hive Tables on a kerberized cluster? ... View more

Thank you for the reply. I switched to using ambari generated certs for all instead. The instructions were helpful. Thank you. ... View more

***Update - I switched back to using Ambari's generated certificate for agents and the server, as I was getting SSL errors related to having the certs not signed by the same 'CA'. Is this because I was using the self-signed certificate for testing locally? I havent' tried this with the CA signed multiple SAN certificate. Also, while comparing the 'Subject Name' on the certificate generated by the Ambari server and the multiple 'Subject Alternative Name' certificate I intended to use originally, the 'Subject name's would have caused discrepancy. Looks like ambari server looks for the node name in the Subject line, but in the SAN certificate I have, the names of the nodes as part of the 'V3 extensions' in the certificate. If you have any suggestions for this scenario, please post. Thanks. ... View more

I tried your steps above, but the ambari-server generates certificates on the agent nodes. To give some context, I have a single certificate with multiple 'subject alternative names' for all the nodes in the cluster. I put that 1 certificate under the /var/lib/ambari-agent/keys folder on all the agents and as soon as I restart Ambari-server, it still does not pick up my '.crt' instead it begins generating the .key, .csr and .crt. My goal is to use the .crt I have to be used by the agent and server on all the nodes for the two_way_ssl functionality. Please advise. ... View more