Cloudera Community

Support Questions

Find answers, ask questions, and share your expertise
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search
Solved Go to solution

Ambari infra fails to start after install on kerberized cluster

avatar
author-rank juan_manuel_nie
Expert Contributor

Created on ‎03-16-2017 08:23 PM - edited ‎08-18-2019 04:38 AM

Hello,

I've upgraded a cluster from 2.4.0 to 2.5.3, and this cluster used Audit to DB for audit, now it's deprecated and I've to use ambari infra.

I've added 2 instances of ambari infra to use solrcloud, and enabled ranger for solrcloud, ranger seems to connect to the zookeeper server and get ambari infra urls, but then it fails to connect on ranger admin start:

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/ranger_admin.py", line 208, in <module>
    RangerAdmin().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280, in execute
    method(env)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 720, in restart
    self.start(env, upgrade_type=upgrade_type)
  File "/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/ranger_admin.py", line 100, in start
    setup_ranger_audit_solr()
  File "/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/setup_ranger_xml.py", line 590, in setup_ranger_audit_solr
    jaas_file = params.solr_jaas_file)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/solr_cloud_util.py", line 116, in create_collection
    Execute(create_collection_cmd)
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 273, in action_run
    tries=self.resource.tries, try_sleep=self.resource.try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
    result = function(command, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
    tries=tries, try_sleep=try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 293, in _call
    raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of 'ambari-sudo.sh JAVA_HOME=/usr/jdk64/jdk1.8.0_60 /usr/lib/ambari-infra-solr-client/solrCloudCli.sh --zookeeper-connect-string hadoop04.int:2181,hadoop03.int:2181,hadoop02.int:2181/infra-solr --create-collection --collection ranger_audits --config-set ranger_audits --shards 1 --replication 1 --max-shards 1 --retry 5 --interval 10 --no-sharding --jaas-file /usr/hdp/current/ranger-admin/conf/ranger_solr_jaas.conf' returned 1. Using default ZkCredentialsProvider
Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
Client environment:host.name=hadoop01.int
Client environment:java.version=1.8.0_60
Client environment:java.vendor=Oracle Corporation
Client environment:java.home=/usr/jdk64/jdk1.8.0_60/jre
Client environment:java.class.path=/usr/lib/ambari-infra-solr-client:/usr/lib/ambari-infra-solr-client/libs/httpclient-4.4.1.jar:/usr/lib/ambari-infra-solr-client/libs/commons-lang-2.5.jar:/usr/lib/ambari-infra-solr-client/libs/slf4j-api-1.7.2.jar:/usr/lib/ambari-infra-solr-client/libs/commons-io-2.1.jar:/usr/lib/ambari-infra-solr-client/libs/easymock-3.4.jar:/usr/lib/ambari-infra-solr-client/libs/junit-4.10.jar:/usr/lib/ambari-infra-solr-client/libs/jcl-over-slf4j-1.7.7.jar:/usr/lib/ambari-infra-solr-client/libs/stax2-api-3.1.4.jar:/usr/lib/ambari-infra-solr-client/libs/jackson-core-asl-1.9.9.jar:/usr/lib/ambari-infra-solr-client/libs/log4j-1.2.17.jar:/usr/lib/ambari-infra-solr-client/libs/ambari-logsearch-solr-client-2.4.1.0.22.jar:/usr/lib/ambari-infra-solr-client/libs/hamcrest-core-1.1.jar:/usr/lib/ambari-infra-solr-client/libs/noggit-0.6.jar:/usr/lib/ambari-infra-solr-client/libs/objenesis-2.2.jar:/usr/lib/ambari-infra-solr-client/libs/slf4j-log4j12-1.7.2.jar:/usr/lib/ambari-infra-solr-client/libs/woodstox-core-asl-4.4.1.jar:/usr/lib/ambari-infra-solr-client/libs/httpmime-4.4.1.jar:/usr/lib/ambari-infra-solr-client/libs/commons-codec-1.8.jar:/usr/lib/ambari-infra-solr-client/libs/commons-cli-1.3.1.jar:/usr/lib/ambari-infra-solr-client/libs/solr-solrj-5.5.2.jar:/usr/lib/ambari-infra-solr-client/libs/jackson-mapper-asl-1.9.13.jar:/usr/lib/ambari-infra-solr-client/libs/zookeeper-3.4.6.jar:/usr/lib/ambari-infra-solr-client/libs/httpcore-4.4.1.jar
Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
Client environment:java.io.tmpdir=/tmp
Client environment:java.compiler=<NA>
Client environment:os.name=Linux
Client environment:os.arch=amd64
Client environment:os.version=2.6.32-573.el6.x86_64
Client environment:user.name=root
Client environment:user.home=/root
Client environment:user.dir=/var/lib/ambari-agent
Initiating client connection, connectString=hadoop04.int:2181,hadoop03.int:2181,hadoop02.int:2181/infra-solr sessionTimeout=15000 watcher=org.apache.solr.common.cloud.SolrZkClient$3@3fb4f649
Waiting for client to connect to ZooKeeper
successfully logged in.
TGT refresh thread started.
Client will use GSSAPI as SASL mechanism.
TGT valid starting at:        Thu Mar 16 18:39:23 CET 2017
TGT expires:                  Fri Mar 17 18:39:23 CET 2017
TGT refresh sleeping until: Fri Mar 17 14:48:54 CET 2017
Opening socket connection to server hadoop04.int/198.18.0.4:2181. Will attempt to SASL-authenticate using Login Context section 'Client'
Socket connection established to hadoop04.int/198.18.0.4:2181, initiating session
Session establishment complete on server hadoop04.int/198.18.0.4:2181, sessionid = 0x35ad7374f89001f, negotiated timeout = 15000
Watcher org.apache.solr.common.cloud.ConnectionManager@bd9131d name:ZooKeeperConnection Watcher:hadoop04.int:2181,hadoop03.int:2181,hadoop02.int:2181/infra-solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
Client is connected to ZooKeeper
Using default ZkACLProvider
Watcher org.apache.solr.common.cloud.ConnectionManager@bd9131d name:ZooKeeperConnection Watcher:hadoop04.int:2181,hadoop03.int:2181,hadoop02.int:2181/infra-solr got event WatchedEvent state:SaslAuthenticated type:None path:null path:null type:None
Setting up SPNego auth with config: /usr/hdp/current/ranger-admin/conf/ranger_solr_jaas.conf
Using default ZkCredentialsProvider
Initiating client connection, connectString=hadoop04.int:2181,hadoop03.int:2181,hadoop02.int:2181/infra-solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.SolrZkClient$3@783e6358
Waiting for client to connect to ZooKeeper
Client will use GSSAPI as SASL mechanism.
Opening socket connection to server hadoop02.int/198.18.0.2:2181. Will attempt to SASL-authenticate using Login Context section 'Client'
Socket connection established to hadoop02.int/198.18.0.2:2181, initiating session
Session establishment complete on server hadoop02.int/198.18.0.2:2181, sessionid = 0x15ad73888170015, negotiated timeout = 10000
Watcher org.apache.solr.common.cloud.ConnectionManager@f9ff59e name:ZooKeeperConnection Watcher:hadoop04.int:2181,hadoop03.int:2181,hadoop02.int:2181/infra-solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
Client is connected to ZooKeeper
Using default ZkACLProvider
Updating cluster state from ZooKeeper... 
Watcher org.apache.solr.common.cloud.ConnectionManager@f9ff59e name:ZooKeeperConnection Watcher:hadoop04.int:2181,hadoop03.int:2181,hadoop02.int:2181/infra-solr got event WatchedEvent state:SaslAuthenticated type:None path:null path:null type:None
No live SolrServers available to handle this request:[http://hadoop02.int:8886/solr, http://hadoop01.int:8886/solr]
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://hadoop02.int:8886/solr, http://hadoop01.int:8886/solr]
 at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:352)
 at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1100)
 at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:870)
 at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:806)
 at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
 at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:166)
 at org.apache.ambari.logsearch.solr.commands.AbstractSolrRetryCommand.createAndProcessRequest(AbstractSolrRetryCommand.java:43)
 at org.apache.ambari.logsearch.solr.commands.AbstractRetryCommand.retry(AbstractRetryCommand.java:45)
 at org.apache.ambari.logsearch.solr.commands.AbstractRetryCommand.run(AbstractRetryCommand.java:40)
 at org.apache.ambari.logsearch.solr.AmbariSolrCloudClient.listCollections(AmbariSolrCloudClient.java:107)
 at org.apache.ambari.logsearch.solr.AmbariSolrCloudClient.createCollection(AmbariSolrCloudClient.java:114)
 at org.apache.ambari.logsearch.solr.AmbariSolrCloudCLI.main(AmbariSolrCloudCLI.java:463)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://hadoop01.int:8886/solr: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<title>Error 404 Not Found</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/admin/collections. Reason:
<pre>    Not Found</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>

</body>
</html>

 at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:545)
 at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
 at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:230)
 at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:372)
 at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:325)
 ... 11 more
No live SolrServers available to handle this request:[http://hadoop02.int:8886/solr, http://hadoop01.int:8886/solr]
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://hadoop02.int:8886/solr, http://hadoop01.int:8886/solr]
 at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:352)
 at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1100)
 at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:870)
 at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:806)
 at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
 at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:166)
 at org.apache.ambari.logsearch.solr.commands.AbstractSolrRetryCommand.createAndProcessRequest(AbstractSolrRetryCommand.java:43)
 at org.apache.ambari.logsearch.solr.commands.AbstractRetryCommand.retry(AbstractRetryCommand.java:45)
 at org.apache.ambari.logsearch.solr.commands.AbstractRetryCommand.run(AbstractRetryCommand.java:40)
 at org.apache.ambari.logsearch.solr.AmbariSolrCloudClient.listCollections(AmbariSolrCloudClient.java:107)
 at org.apache.ambari.logsearch.solr.AmbariSolrCloudClient.createCollection(AmbariSolrCloudClient.java:114)
 at org.apache.ambari.logsearch.solr.AmbariSolrCloudCLI.main(AmbariSolrCloudCLI.java:463)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://hadoop01.int:8886/solr: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<title>Error 404 Not Found</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/admin/collections. Reason:
<pre>    Not Found</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>

</body>
</html>

 at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:545)
 at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
 at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:230)
 at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:372)
 at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:325)
 ... 11 more
Command failed, tries again (tries: 1)
No live SolrServers available to handle this request:[http://hadoop02.int:8886/solr, http://hadoop01.int:8886/solr]
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://hadoop02.int:8886/solr, http://hadoop01.int:8886/solr]
 at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:352)
 at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1100)
 at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:870)
 at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:806)
 at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
 at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:166)
 at org.apache.ambari.logsearch.solr.commands.AbstractSolrRetryCommand.createAndProcessRequest(AbstractSolrRetryCommand.java:43)
 at org.apache.ambari.logsearch.solr.commands.AbstractRetryCommand.retry(AbstractRetryCommand.java:45)
 at org.apache.ambari.logsearch.solr.commands.AbstractRetryCommand.retry(AbstractRetryCommand.java:54)
 at org.apache.ambari.logsearch.solr.commands.AbstractRetryCommand.run(AbstractRetryCommand.java:40)
 at org.apache.ambari.logsearch.solr.AmbariSolrCloudClient.listCollections(AmbariSolrCloudClient.java:107)
 at org.apache.ambari.logsearch.solr.AmbariSolrCloudClient.createCollection(AmbariSolrCloudClient.java:114)
 at org.apache.ambari.logsearch.solr.AmbariSolrCloudCLI.main(AmbariSolrCloudCLI.java:463)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://hadoop01.int:8886/solr: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<title>Error 404 Not Found</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/admin/collections. Reason:
<pre>    Not Found</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>

</body>
</html>

 at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:545)
 at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
 at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:230)
 at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:372)
 at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:341)
 ... 12 more
No live SolrServers available to handle this request:[http://hadoop02.int:8886/solr, http://hadoop01.int:8886/solr]
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://hadoop02.int:8886/solr, http://hadoop01.int:8886/solr]
 at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:352)
 at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1100)
 at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:870)
 at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:806)
 at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
 at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:166)
 at org.apache.ambari.logsearch.solr.commands.AbstractSolrRetryCommand.createAndProcessRequest(AbstractSolrRetryCommand.java:43)
 at org.apache.ambari.logsearch.solr.commands.AbstractRetryCommand.retry(AbstractRetryCommand.java:45)
 at org.apache.ambari.logsearch.solr.commands.AbstractRetryCommand.retry(AbstractRetryCommand.java:54)
 at org.apache.ambari.logsearch.solr.commands.AbstractRetryCommand.run(AbstractRetryCommand.java:40)
 at org.apache.ambari.logsearch.solr.AmbariSolrCloudClient.listCollections(AmbariSolrCloudClient.java:107)
 at org.apache.ambari.logsearch.solr.AmbariSolrCloudClient.createCollection(AmbariSolrCloudClient.java:114)
 at org.apache.ambari.logsearch.solr.AmbariSolrCloudCLI.main(AmbariSolrCloudCLI.java:463)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://hadoop01.int:8886/solr: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<title>Error 404 Not Found</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/admin/collections. Reason:
<pre>    Not Found</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>

</body>
</html>

 at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:545)
 at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
 at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:230)
 at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:372)
 at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:341)
 ... 12 more
Command failed, tries again (tries: 2)
No live SolrServers available to handle this request:[http://hadoop02.int:8886/solr, http://hadoop01.int:8886/solr]
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://hadoop02.int:8886/solr, http://hadoop01.int:8886/solr]
 at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:352)
 at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1100)
 at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:870)
 at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:806)
 at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
 at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:166)
 at org.apache.ambari.logsearch.solr.commands.AbstractSolrRetryCommand.createAndProcessRequest(AbstractSolrRetryCommand.java:43)
 at org.apache.ambari.logsearch.solr.commands.AbstractRetryCommand.retry(AbstractRetryCommand.java:45)
 at org.apache.ambari.logsearch.solr.commands.AbstractRetryCommand.retry(AbstractRetryCommand.java:54)
 at org.apache.ambari.logsearch.solr.commands.AbstractRetryCommand.retry(AbstractRetryCommand.java:54)
 at org.apache.ambari.logsearch.solr.commands.AbstractRetryCommand.run(AbstractRetryCommand.java:40)
 at org.apache.ambari.logsearch.solr.AmbariSolrCloudClient.listCollections(AmbariSolrCloudClient.java:107)
 at org.apache.ambari.logsearch.solr.AmbariSolrCloudClient.createCollection(AmbariSolrCloudClient.java:114)
 at org.apache.ambari.logsearch.solr.AmbariSolrCloudCLI.main(AmbariSolrCloudCLI.java:463)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://hadoop01.int:8886/solr: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<title>Error 404 Not Found</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/admin/collections. Reason:
<pre>    Not Found</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>

</body>
</html>

 at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:545)
 at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
 at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:230)
 at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:372)
 at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:341)
 ... 13 more
No live SolrServers available to handle this request:[http://hadoop02.int:8886/solr, http://hadoop01.int:8886/solr]
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://hadoop02.int:8886/solr, http://hadoop01.int:8886/solr]
 at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:352)
 at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1100)
 at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:870)
 at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:806)
 at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
 at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:166)
 at org.apache.ambari.logsearch.solr.commands.AbstractSolrRetryCommand.createAndProcessRequest(AbstractSolrRetryCommand.java:43)
 at org.apache.ambari.logsearch.solr.commands.AbstractRetryCommand.retry(AbstractRetryCommand.java:45)
 at org.apache.ambari.logsearch.solr.commands.AbstractRetryCommand.retry(AbstractRetryCommand.java:54)
 at org.apache.ambari.logsearch.solr.commands.AbstractRetryCommand.retry(AbstractRetryCommand.java:54)
 at org.apache.ambari.logsearch.solr.commands.AbstractRetryCommand.run(AbstractRetryCommand.java:40)
 at org.apache.ambari.logsearch.solr.AmbariSolrCloudClient.listCollections(AmbariSolrCloudClient.java:107)
 at org.apache.ambari.logsearch.solr.AmbariSolrCloudClient.createCollection(AmbariSolrCloudClient.java:114)
 at org.apache.ambari.logsearch.solr.AmbariSolrCloudCLI.main(AmbariSolrCloudCLI.java:463)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://hadoop01.int:8886/solr: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<title>Error 404 Not Found</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/admin/collections. Reason:
<pre>    Not Found</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>

</body>
</html>

 at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:545)
 at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
 at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:230)
 at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:372)
 at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:341)
 ... 13 more
Command failed, tries again (tries: 3)
No live SolrServers available to handle this request:[http://hadoop02.int:8886/solr, http://hadoop01.int:8886/solr]
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://hadoop02.int:8886/solr, http://hadoop01.int:8886/solr]
 at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:352)
 at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1100)
 at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:870)
 at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:806)
 at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
 at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:166)
 at org.apache.ambari.logsearch.solr.commands.AbstractSolrRetryCommand.createAndProcessRequest(AbstractSolrRetryCommand.java:43)
 at org.apache.ambari.logsearch.solr.commands.AbstractRetryCommand.retry(AbstractRetryCommand.java:45)
 at org.apache.ambari.logsearch.solr.commands.AbstractRetryCommand.retry(AbstractRetryCommand.java:54)
 at org.apache.ambari.logsearch.solr.commands.AbstractRetryCommand.retry(AbstractRetryCommand.java:54)
 at org.apache.ambari.logsearch.solr.commands.AbstractRetryCommand.retry(AbstractRetryCommand.java:54)
 at org.apache.ambari.logsearch.solr.commands.AbstractRetryCommand.run(AbstractRetryCommand.java:40)
 at org.apache.ambari.logsearch.solr.AmbariSolrCloudClient.listCollections(AmbariSolrCloudClient.java:107)
 at org.apache.ambari.logsearch.solr.AmbariSolrCloudClient.createCollection(AmbariSolrCloudClient.java:114)
 at org.apache.ambari.logsearch.solr.AmbariSolrCloudCLI.main(AmbariSolrCloudCLI.java:463)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://hadoop01.int:8886/solr: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<title>Error 404 Not Found</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/admin/collections. Reason:
<pre>    Not Found</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>

</body>
</html>

 at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:545)
 at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
 at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:230)
 at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:372)
 at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:341)
 ... 14 more
No live SolrServers available to handle this request:[http://hadoop02.int:8886/solr, http://hadoop01.int:8886/solr]
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://hadoop02.int:8886/solr, http://hadoop01.int:8886/solr]
 at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:352)
 at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1100)
 at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:870)
 at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:806)
 at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
 at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:166)
 at org.apache.ambari.logsearch.solr.commands.AbstractSolrRetryCommand.createAndProcessRequest(AbstractSolrRetryCommand.java:43)
 at org.apache.ambari.logsearch.solr.commands.AbstractRetryCommand.retry(AbstractRetryCommand.java:45)
 at org.apache.ambari.logsearch.solr.commands.AbstractRetryCommand.retry(AbstractRetryCommand.java:54)
 at org.apache.ambari.logsearch.solr.commands.AbstractRetryCommand.retry(AbstractRetryCommand.java:54)
 at org.apache.ambari.logsearch.solr.commands.AbstractRetryCommand.retry(AbstractRetryCommand.java:54)
 at org.apache.ambari.logsearch.solr.commands.AbstractRetryCommand.run(AbstractRetryCommand.java:40)
 at org.apache.ambari.logsearch.solr.AmbariSolrCloudClient.listCollections(AmbariSolrCloudClient.java:107)
 at org.apache.ambari.logsearch.solr.AmbariSolrCloudClient.createCollection(AmbariSolrCloudClient.java:114)
 at org.apache.ambari.logsearch.solr.AmbariSolrCloudCLI.main(AmbariSolrCloudCLI.java:463)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://hadoop01.int:8886/solr: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<title>Error 404 Not Found</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/admin/collections. Reason:
<pre>    Not Found</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>

</body>
</html>

 at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:545)
 at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
 at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:230)
 at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:372)
 at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:341)
 ... 14 more
Command failed, tries again (tries: 4)
No live SolrServers available to handle this request:[http://hadoop02.int:8886/solr, http://hadoop01.int:8886/solr]
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://hadoop02.int:8886/solr, http://hadoop01.int:8886/solr]
 at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:352)
 at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1100)
 at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:870)
 at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:806)
 at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
 at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:166)
 at org.apache.ambari.logsearch.solr.commands.AbstractSolrRetryCommand.createAndProcessRequest(AbstractSolrRetryCommand.java:43)
 at org.apache.ambari.logsearch.solr.commands.AbstractRetryCommand.retry(AbstractRetryCommand.java:45)
 at org.apache.ambari.logsearch.solr.commands.AbstractRetryCommand.retry(AbstractRetryCommand.java:54)
 at org.apache.ambari.logsearch.solr.commands.AbstractRetryCommand.retry(AbstractRetryCommand.java:54)
 at org.apache.ambari.logsearch.solr.commands.AbstractRetryCommand.retry(AbstractRetryCommand.java:54)
 at org.apache.ambari.logsearch.solr.commands.AbstractRetryCommand.retry(AbstractRetryCommand.java:54)
 at org.apache.ambari.logsearch.solr.commands.AbstractRetryCommand.run(AbstractRetryCommand.java:40)
 at org.apache.ambari.logsearch.solr.AmbariSolrCloudClient.listCollections(AmbariSolrCloudClient.java:107)
 at org.apache.ambari.logsearch.solr.AmbariSolrCloudClient.createCollection(AmbariSolrCloudClient.java:114)
 at org.apache.ambari.logsearch.solr.AmbariSolrCloudCLI.main(AmbariSolrCloudCLI.java:463)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://hadoop01.int:8886/solr: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<title>Error 404 Not Found</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/admin/collections. Reason:
<pre>    Not Found</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>

</body>
</html>

 at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:545)
 at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
 at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:230)
 at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:372)
 at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:341)
 ... 15 more
No live SolrServers available to handle this request:[http://hadoop02.int:8886/solr, http://hadoop01.int:8886/solr]
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://hadoop02.int:8886/solr, http://hadoop01.int:8886/solr]
 at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:352)
 at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1100)
 at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:870)
 at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:806)
 at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
 at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:166)
 at org.apache.ambari.logsearch.solr.commands.AbstractSolrRetryCommand.createAndProcessRequest(AbstractSolrRetryCommand.java:43)
 at org.apache.ambari.logsearch.solr.commands.AbstractRetryCommand.retry(AbstractRetryCommand.java:45)
 at org.apache.ambari.logsearch.solr.commands.AbstractRetryCommand.retry(AbstractRetryCommand.java:54)
 at org.apache.ambari.logsearch.solr.commands.AbstractRetryCommand.retry(AbstractRetryCommand.java:54)
 at org.apache.ambari.logsearch.solr.commands.AbstractRetryCommand.retry(AbstractRetryCommand.java:54)
 at org.apache.ambari.logsearch.solr.commands.AbstractRetryCommand.retry(AbstractRetryCommand.java:54)
 at org.apache.ambari.logsearch.solr.commands.AbstractRetryCommand.run(AbstractRetryCommand.java:40)
 at org.apache.ambari.logsearch.solr.AmbariSolrCloudClient.listCollections(AmbariSolrCloudClient.java:107)
 at org.apache.ambari.logsearch.solr.AmbariSolrCloudClient.createCollection(AmbariSolrCloudClient.java:114)
 at org.apache.ambari.logsearch.solr.AmbariSolrCloudCLI.main(AmbariSolrCloudCLI.java:463)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://hadoop01.int:8886/solr: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<title>Error 404 Not Found</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/admin/collections. Reason:
<pre>    Not Found</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>

</body>
</html>

 at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:545)
 at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
 at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:230)
 at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:372)
 at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:341)
 ... 15 more
Command failed, tries again (tries: 5)
usage:
./solrCloudCli.sh --create-collection -z host1:2181,host2:2181/ambari-solr -c collection -cs conf_set
./solrCloudCli.sh --upload-config -z host1:2181,host2:2181/ambari-solr -d /tmp/myconfig_dir -cs config_set
./solrCloudCli.sh --download-config -z host1:2181,host2:2181/ambari-solr -cs config_set -d /tmp/myonfig_dir
./solrCloudCli.sh --check-config -z host1:2181,host2:2181/ambari-solr -cs config_set
./solrCloudCli.sh --create-shard -z host1:2181,host2:2181/ambari-solr -c collection -sn myshard
./solrCloudCli.sh --create-znode -z host1:2181,host2:2181 -zn /ambari-solr
./solrCloudCli.sh --check-znode -z host1:2181,host2:2181 -zn /ambari-solr
./solrCloudCli.sh --cluster-prop -z host1:2181,host2:2181/ambari-solr -cpn urlScheme -cpn http
./solrCloudCli.sh --create-sasl-users -z host1:2181,host2:2181 -zn /ambari-solr -csu logsearch,atlas,ranger
./solrCloudCli.sh --setup-kerberos -z host1:2181,host2:2181 --secure -zn /ambari-solr-secure -cfz /ambari-solr-unsecure -jf /etc/path/my_jaas.conf
./solrCloudCli.sh --setup-kerberos-plugin -z host1:2181,host2:2181 -zn /ambari-solr
 -c,--collection <collection name>                                          Collection name
 -cc,--create-collection                                                    Create collection in Solr (command)
 -cfz,--copy-from-znode </ambari-solr-secure>                               Copy-from-znode
 -chc,--check-config                                                        Check configuration exists in Zookeeper (command)
 -chz,--check-znode                                                         Check znode exists in Zookeeper (command)
 -cp,--cluster-prop                                                         Set cluster property (command)
 -cpn,--property-name <cluster prop name>                                   Cluster property name
 -cpv,--property-value <cluster prop value>                                 Cluster property value
 -cs,--config-set <config_set>                                              Configuration set
 -csh,--create-shard                                                        Create shard in Solr (command)
 -csu,--create-sasl-users                                                   Create sasl users
 -cz,--create-znode                                                         Create Znode (command)
 -d,--config-dir <config_dir>                                               Configuration directory
 -dc,--download-config                                                      Download configuration set from Zookeeper (command)
 -h,--help                                                                  Print commands
 -i,--interval <interval>                                                   Interval for retry logic in sec [default:5]
 -jf,--jaas-file <jaas_file>                                                Location of the jaas-file to communicate with kerberized Solr
 -ksl,--key-store-location <key store location>                             Location of the key store used to communicate with Solr using SSL
 -ksp,--key-store-password <key store password>                             Key store password used to communicate with Solr using SSL
 -kst,--key-store-type <key store type>                                     Type of the key store used to communicate with Solr using SSL
 -m,--max-shards <max number of shards>                                     Max number of shards per node (default: replication * shards)
 -ns,--no-sharding                                                          Sharding not used when creating collection
 -r,--replication <replication factor>                                      Replication factor
 -rf,--router-field <router_field>                                          Router field for collection [default:_router_field_]
 -rn,--router-name <router_name>                                            Router name for collection [default:implicit]
 -rt,--retry <number of retries>                                            Number of retries for access Solr [default:10]
 -s,--shards <shard number>                                                 Number of shards
 -sec,--secure                                                              Flag for enable/disable kerberos (with --setup-kerberos or --setup-kerberos-plugin)
 -sk,--setup-kerberos                                                       Setup kerberos (command)
 -skp,--setup-kerberos-plugin                                               Setup kerberos plugin in security.json (command)
 -sn,--shard-name <my_new_shard>                                            Name of the shard for create-shard command
 -su,--sasl-users <atlas,ranger,logsearch-solr>                             Sasl users (comma separated list)
 -tsl,--trust-store-location <trust store location>                         Location of the trust store used to communicate with Solr using SSL
 -tsp,--trust-store-password <trust store password>                         Trust store password used to communicate with Solr using SSL
 -tst,--trust-store-type <trust store type>                                 Type of the trust store used to communicate with Solr using SSL
 -uc,--upload-config                                                        Upload configuration set to Zookeeper (command)
 -z,--zookeeper-connect-string <host:port,host:port[/ambari-solr]>          Zookeeper quorum [and Znode (optional)]
 -zn,--znode </ambari-solr>                                                 Zookeeper ZNode
Maximum retries exceeded: 5
Maximum retries exceeded: 5
Return code: 1

Ambari shows both instances of ambari infra as running but the solr.log have the following:

[root@hadoop01 ambari-infra-solr]# cat solr.log
2017-03-16 18:23:42,576 [main] WARN  [   ] org.eclipse.jetty.server.handler.RequestLogHandler (RequestLogHandler.java:137) - !RequestLog
2017-03-16 18:23:43,310 [main] WARN  [   ] org.eclipse.jetty.security.ConstraintSecurityHandler (ConstraintSecurityHandler.java:807) - ServletContext@o.e.j.w.WebAppContext@544fe44c{/solr,file:/usr/lib/ambari-infra-solr/server/solr-webapp/webapp/,STARTING}{/usr/lib/ambari-infra-solr/server/solr-webapp/webapp} has uncovered http methods for path: /
2017-03-16 18:23:43,646 [main] WARN  [   ] org.apache.solr.core.CoreContainer (CoreContainer.java:398) - Couldn't add files from /opt/ambari_infra_solr/data/lib to classpath: /opt/ambari_infra_solr/data/lib
2017-03-16 18:23:44,264 [main] ERROR [   ] org.apache.solr.servlet.SolrDispatchFilter (SolrDispatchFilter.java:141) - Could not start Solr. Check solr/home property and the logs
2017-03-16 18:23:44,297 [main] ERROR [   ] org.apache.solr.common.SolrException (SolrException.java:159) - null:org.apache.solr.common.SolrException: Missing required parameter 'solr.kerberos.principal'.
        at org.apache.solr.security.KerberosPlugin.putParam(KerberosPlugin.java:135)
        at org.apache.solr.security.KerberosPlugin.init(KerberosPlugin.java:82)
        at org.apache.solr.core.CoreContainer.initializeAuthenticationPlugin(CoreContainer.java:292)
        at org.apache.solr.core.CoreContainer.load(CoreContainer.java:418)
        at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:158)
        at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:134)
        at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:138)
        at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:852)
        at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:298)
        at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1349)
        at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342)
        at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:741)
        at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:505)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:41)
        at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:186)
        at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:498)
        at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:146)
        at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:180)
        at org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProvider.java:461)
        at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:64)
        at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:609)
        at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:528)
        at org.eclipse.jetty.util.Scanner.scan(Scanner.java:391)
        at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:313)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:150)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:560)
        at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:235)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
        at org.eclipse.jetty.server.Server.start(Server.java:387)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
        at org.eclipse.jetty.server.Server.doStart(Server.java:354)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1255)
        at java.security.AccessController.doPrivileged(Native Method)
        at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1174)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.eclipse.jetty.start.Main.invokeMain(Main.java:321)
        at org.eclipse.jetty.start.Main.start(Main.java:817)
        at org.eclipse.jetty.start.Main.main(Main.java:112)

I've check the configuration and seems to be ok, I just found that solr was using spnego keytabs and changed the variables to solr-infra keytabs but the error persist.

If I try to access the webui I can using the spnego ticket of my AD without any problem, but the console holds loading

13702-prtscr-capture.jpg

Any clue about what is going on?

Thank you in advance

7,746 Views
0 Kudos
1 ACCEPTED SOLUTION

avatar
author-rank juan_manuel_nie
Expert Contributor

Created ‎03-21-2017 11:35 AM

I finally solved this.

Everything was fine but I was hitting https://issues.apache.org/jira/browse/AMBARI-18898

So I solved this doing the following:

  • Quoted all $ in Ambari Infra > Configs > Advanced > Infra Solr Kerberos name rules
  • In Ambari Infra > Configs > Advanced > infra-solr-env template quoted the value of SOLR_KERB_NAME_RULES:
SOLR_KERB_NAME_RULES="{{infra_solr_kerberos_name_rules}}"
  • In Ambari Infra > Configs > Advanced > infra-solr-env template removed the value -Dsolr.kerberos.name.rules="$SOLR_KERB_NAME_RULES" from SOLR_AUTHENTICATION_OPTS resulting as follows:
SOLR_AUTHENTICATION_OPTS=" -DauthenticationPlugin=org.apache.solr.security.KerberosPlugin -Djava.security.auth.login.config=$SOLR_JAAS_FILE -Dsolr.kerberos.principal=${SOLR_KERB_PRINCIPAL} -Dsolr.kerberos.keytab=${SOLR_KERB_KEYTAB} -Dsolr.kerberos.cookie.domain=${SOLR_HOST} "
  • In each solr server modify /usr/lib/ambari-infra-solr/bin/solr and add -Dsolr.kerberos.name.rules=$SOLR_KERB_NAME_RULES to SOLR_START_OPTS variable:
SOLR_START_OPTS=('-server' "${JAVA_MEM_OPTS[@]}" "${GC_TUNE[@]}" "${GC_LOG_OPTS[@]}" \
    "${REMOTE_JMX_OPTS[@]}" "${CLOUD_MODE_OPTS[@]}" \
    "-Djetty.port=$SOLR_PORT" "-DSTOP.PORT=$stop_port" "-DSTOP.KEY=$STOP_KEY" \
    "${SOLR_HOST_ARG[@]}" "-Duser.timezone=$SOLR_TIMEZONE" \
    "-Djetty.home=$SOLR_SERVER_DIR" "-Dsolr.solr.home=$SOLR_HOME" "-Dsolr.install.dir=$SOLR_TIP" \
    "${LOG4J_CONFIG[@]}" "${SOLR_OPTS[@]}" -Dsolr.kerberos.name.rules="$SOLR_KERB_NAME_RULES")

After all this modifications just start ambari infra and it will start properly :).

View solution in original post

5,890 Views
6 REPLIES 6

avatar
author-rank pminovic author-rank
Master Guru

Created ‎03-17-2017 01:41 AM

You need 2 principals and keytabs per instance: SOLR_KERBEROS_PRINCIPAL=HTTP/solr-host1.fqdn@YOUR.REALM, this will be saved in infra-solr-env.sh, and principal=infra-solr/solr-host1.fqdn@YOUR.REALM, this one will go to infra_solr_jaas.conf. And two more for your host2. You can set both from Ambari, in the "Advanced infra-solr-env" section, referring to the host fqdn as _HOST, for example HTTP/_HOST@YOUR.REALM.

5,890 Views
0 Kudos

avatar
author-rank juan_manuel_nie
Expert Contributor

Created ‎03-17-2017 10:32 AM

@Predrag Minovic

Hello, I dont have exactly that variable, i have the following in myambari:

SOLR_KERB_PRINCIPAL={{infra_solr_web_kerberos_principal}}
....

SOLR_AUTHENTICATION_OPTS=" -DauthenticationPlugin=org.apache.solr.security.KerberosPlugin -Djava.security.auth.login.config=$SOLR_JAAS_FILE -Dsolr.kerberos.principal=${SOLR_KERB_PRINCIPAL} -Dsolr.kerberos.keytab=${SOLR_KERB_KEYTAB} -Dsolr.kerberos.cookie.domain=${SOLR_HOST} -Dsolr.kerberos.name.rules=${SOLR_KERB_NAME_RULES}"

The {{infra_solr_web_kerberos_principal}} it's HTTP/_HOST@HADOOP.INT

And the infra_solr_jaas.conf generated by ambari is correctly pointing in each server it's principal

hadoop01:

[root@hadoop01 ~]# cat /etc/ambari-infra-solr/conf/infra_solr_jaas.conf
Client {
 com.sun.security.auth.module.Krb5LoginModule required
 useKeyTab=true
 storeKey=true
 useTicketCache=false
 keyTab="/etc/security/keytabs/ambari-infra-solr.service.keytab"
 principal="infra-solr/hadoop01.int@HADOOP.INT";

hadoop02:

[root@hadoop02 ~]# cat /etc/ambari-infra-solr/conf/infra_solr_jaas.conf
Client {
 com.sun.security.auth.module.Krb5LoginModule required
 useKeyTab=true
 storeKey=true
 useTicketCache=false
 keyTab="/etc/security/keytabs/ambari-infra-solr.service.keytab"
 principal="infra-solr/hadoop02.int@HADOOP.INT";

So I don't know what maybe the problem, should I add declare SOLR_KERBEROS_PRINCIPAL?

5,890 Views
0 Kudos

avatar
author-rank aleekha
Rising Star

Created ‎03-18-2017 09:01 PM

Hi @Juan Manuel Nieto: You need to regenerate keytabs after the upgrade completes. [Ambari UI->Admin->Kerberos->regenerate key tabs btn] Else Ranger Admin will fail to start.

5,890 Views

avatar
author-rank juan_manuel_nie
Expert Contributor

Created ‎03-21-2017 10:17 AM

Hello @Anuja Leekha

I have regenerated the keytab and still having the same errors:

errors-12524.txt

Can this be happening because I had installed ranger before install ambari infra?

Preview file
39 KB
5,890 Views
0 Kudos

avatar
author-rank juan_manuel_nie
Expert Contributor

Created ‎03-21-2017 11:35 AM

I finally solved this.

Everything was fine but I was hitting https://issues.apache.org/jira/browse/AMBARI-18898

So I solved this doing the following:

  • Quoted all $ in Ambari Infra > Configs > Advanced > Infra Solr Kerberos name rules
  • In Ambari Infra > Configs > Advanced > infra-solr-env template quoted the value of SOLR_KERB_NAME_RULES:
SOLR_KERB_NAME_RULES="{{infra_solr_kerberos_name_rules}}"
  • In Ambari Infra > Configs > Advanced > infra-solr-env template removed the value -Dsolr.kerberos.name.rules="$SOLR_KERB_NAME_RULES" from SOLR_AUTHENTICATION_OPTS resulting as follows:
SOLR_AUTHENTICATION_OPTS=" -DauthenticationPlugin=org.apache.solr.security.KerberosPlugin -Djava.security.auth.login.config=$SOLR_JAAS_FILE -Dsolr.kerberos.principal=${SOLR_KERB_PRINCIPAL} -Dsolr.kerberos.keytab=${SOLR_KERB_KEYTAB} -Dsolr.kerberos.cookie.domain=${SOLR_HOST} "
  • In each solr server modify /usr/lib/ambari-infra-solr/bin/solr and add -Dsolr.kerberos.name.rules=$SOLR_KERB_NAME_RULES to SOLR_START_OPTS variable:
SOLR_START_OPTS=('-server' "${JAVA_MEM_OPTS[@]}" "${GC_TUNE[@]}" "${GC_LOG_OPTS[@]}" \
    "${REMOTE_JMX_OPTS[@]}" "${CLOUD_MODE_OPTS[@]}" \
    "-Djetty.port=$SOLR_PORT" "-DSTOP.PORT=$stop_port" "-DSTOP.KEY=$STOP_KEY" \
    "${SOLR_HOST_ARG[@]}" "-Duser.timezone=$SOLR_TIMEZONE" \
    "-Djetty.home=$SOLR_SERVER_DIR" "-Dsolr.solr.home=$SOLR_HOME" "-Dsolr.install.dir=$SOLR_TIP" \
    "${LOG4J_CONFIG[@]}" "${SOLR_OPTS[@]}" -Dsolr.kerberos.name.rules="$SOLR_KERB_NAME_RULES")

After all this modifications just start ambari infra and it will start properly :).

5,891 Views

avatar
author-rank pminovic author-rank
Master Guru

Created ‎03-22-2017 08:24 AM

Hi @Juan Manuel Nieto, well done! I noticed AMBARI-18898 and suspected it's causing a havoc on the command line, but didn't have time to try it. Though, now, after fixing Solr, Ranger audit cannot connect to it and Ambari is showing "http 500" (false) alerts on both Infra Solr instances. Edit: I missed "DEFAULT" in the name rules, omitted as I tried with only one rule before. After adding DEFAULT everything is back to normal!

5,890 Views
0 Kudos
Announcements
What's New @ Cloudera
Cloudera Data Engineering 1.23: Access Spark from Your Favor...
What's New @ Cloudera
HBase REST server scaling support is Generally Available
What's New @ Cloudera
New CLI option in the update-database command
What's New @ Cloudera
New Action menu item in the Cloudera Operational Database UI
What's New @ Cloudera
Get the list of supported instance types in the Cloudera Ope...
View More Announcements