Cloudera Community

Welcome to the Cloudera Community

cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Who agreed with this topic

Solr Bad Health when using Sentry in Quickstart VM

avatar
author-rank dunnej
Contributor

Created on ‎11-22-2016 02:47 PM - edited ‎11-22-2016 04:22 PM

Background:

 

I have been experimenting with CDH 5.8 in the Quickstart VM toward some initial evaluation work.  I wanted to explore Sentry related features in the VM.  I setup up Kerberos locally in the VM and installed the Sentry service to the VM cluster.  I skipped setting up recommended TLS Encryption since I don't require hardened security configurations for my functional testing.  After enabling Sentry service in Solr, I immediately began seeing problems with Solr's health tests.

 

I was able to do my functional review of Sentry based security features in CDH and just skipped evaluating Solr's security features.  I then wanted to play with HDFS Transparent Encryption so I enabled TLS Level 3 Encryption.  After I completed my HDFS Encryption evaluation, I decided to go back to the Solr issue to see if it was somehow related to TLS encryption not being enabled.  When I enabled Sentry service for Solr, it still had bad health issues.

 

I'm now curious to debug the Solr problem further, so I'm starting this thread. 

 

Here's a summary of the current behaviour with TLS level 3, Local Kerberos, and Sentry service installed:

 

When I start Solr with "Sentry Service" Solr config value set to "none", restart the service, then Solr starts successfully and remains in good health.

 

If I change that setting to "Sentry" for Solr, restart the service, then Solr starts up initially in good health, but once all the health tests become enabled, then it immediately goes Red for Bad Heatlh with the "Solr Server API Liveness" and "Web Server Status" tests failling.  The Solr parent process is still running and in good health, but the Solr server itself has died.  When I look at /var/log/solr/solr-cmf-solr-SOLR_SERVER-quickstart.cloudera.log.out, there are some log entries that are not observed in the non-Sentry behaviour above.  The suspicious log fragments follow below:

2016-11-22 12:37:43,405 ERROR org.apache.solr.servlet.SolrDispatchFilter: Could not start Solr. Check solr/home property and the logs
2016-11-22 12:37:43,428 INFO org.apache.solr.cloud.Overseer: Starting to work on the main queue
2016-11-22 12:37:43,476 INFO org.apache.solr.cloud.Overseer: DownNode state invoked for node: quickstart.cloudera:8983_solr
2016-11-22 12:37:43,482 ERROR org.apache.solr.core.SolrCore: null:java.lang.NoSuchFieldError: LISTSNAPSHOTS
	at org.apache.solr.handler.admin.SecureCollectionsHandler.<clinit>(SecureCollectionsHandler.java:38)
<full stacktrace snipped -  see full logs below>

 

At first I thought this might be a red herring but since the origin of the stacktrace is SecureCollectionsHandler, then I thought it could be the root cause after all.

 

Steps to rule out obvious causes:

 

1) I thought it might be related to some indices I had created as part of working through the tutorial.  To rule this out I disabled Sentry and started up Solr, then removed all existing indices and dependant dashboards via Hue UI.  After enabling Sentry for Solr and restarting, the behaviour was unchanged.

 

2) I searched around for anything to do with the specific error or snapshots in general.  The only thing that I found was that there is an HDFS snapshot feature.  As a long shot I thought it might be a bug where Solr was never tested with Sentry enabled WITHOUT there being HDFS snapshots present.  I enabled HDFS snapshots and took a few while Solr was running with Sentry disabled.  I then re-enabled Sentry for Solr and restart to find same behavious with LISTSNAPSHOTS error.

 

3) Memory issues.  I've had problems when turning on TLS in Quickstart VM where there was strange behaviour in Cloudera Manager with Service Roles dying without obvious cause.  I ultimately tracked them to extremely low heap sizes being configured below minimum recommendations for manager service roles.  In those instances the behaviour was fixed by increasing the memory to affected services to the minimum recommendation.  I decided to try this with Solr despite it already being allocated at the minimal levels, so I doubled the heap and total memory allocations for Solr and cold-restarted the service.  The bad health behaviour and the LISTSNAPSHOTS error continued.

 

Any ideas?!

 

Finally my curiousity got the best of me and I decided to post here to see if anyone has any ideas on how to resolve or at least debug it further.

 

I would like to be able to explore Solr from a security point of view in the QuickStart VM if possible!

 

Can anyone help?

 

Full Solr logs (/var/log/solr/solr-cmf-solr-SOLR_SERVER-quickstart.cloudera.log.out) follow below:

 

Without Sentry Enabled:

2016-11-22 11:54:00,305 INFO org.apache.solr.core.SolrResourceLoader: No /solr/home in JNDI
2016-11-22 11:54:00,306 INFO org.apache.solr.core.SolrResourceLoader: using system property solr.solr.home: /var/lib/solr
2016-11-22 11:54:00,307 INFO org.apache.solr.core.SolrResourceLoader: new SolrResourceLoader for directory: '/var/lib/solr/'
2016-11-22 11:54:00,540 INFO org.apache.solr.servlet.SolrDispatchFilter: Trying to read solr.xml from quickstart.cloudera:2181/solr
2016-11-22 11:54:00,554 INFO org.apache.solr.common.cloud.SolrZkClient: Using default ZkCredentialsProvider
2016-11-22 11:54:00,568 INFO org.apache.zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.5-cdh5.8.0--1, built on 06/16/2016 19:37 GMT
2016-11-22 11:54:00,568 INFO org.apache.zookeeper.ZooKeeper: Client environment:host.name=quickstart.cloudera
2016-11-22 11:54:00,568 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.version=1.7.0_67
2016-11-22 11:54:00,568 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
2016-11-22 11:54:00,568 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.home=/usr/java/jdk1.7.0_67-cloudera/jre
2016-11-22 11:54:00,568 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.class.path=/usr/lib/bigtop-tomcat/bin/bootstrap.jar
2016-11-22 11:54:00,569 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2016-11-22 11:54:00,569 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/var/lib/solr/
2016-11-22 11:54:00,569 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
2016-11-22 11:54:00,569 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.name=Linux
2016-11-22 11:54:00,569 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.arch=amd64
2016-11-22 11:54:00,569 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.version=2.6.32-573.el6.x86_64
2016-11-22 11:54:00,569 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.name=solr
2016-11-22 11:54:00,569 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.home=/var/lib/solr
2016-11-22 11:54:00,569 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.dir=/var/run/cloudera-scm-agent/process/373-solr-SOLR_SERVER
2016-11-22 11:54:00,570 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=quickstart.cloudera:2181/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@497fec5
2016-11-22 11:54:00,588 INFO org.apache.solr.common.cloud.ConnectionManager: Waiting for client to connect to ZooKeeper
2016-11-22 11:54:00,738 INFO org.apache.zookeeper.Login: successfully logged in.
2016-11-22 11:54:00,746 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2016-11-22 11:54:00,751 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server quickstart.cloudera/10.0.2.15:2181. Will attempt to SASL-authenticate using Login Context section 'Client'
2016-11-22 11:54:00,752 INFO org.apache.zookeeper.Login: TGT refresh thread started.
2016-11-22 11:54:00,755 INFO org.apache.zookeeper.Login: TGT valid starting at:        Tue Nov 22 11:54:00 PST 2016
2016-11-22 11:54:00,755 INFO org.apache.zookeeper.Login: TGT expires:                  Wed Nov 23 11:54:00 PST 2016
2016-11-22 11:54:00,756 INFO org.apache.zookeeper.Login: TGT refresh sleeping until: Wed Nov 23 07:51:50 PST 2016
2016-11-22 11:54:00,761 INFO org.apache.zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.0.2.15:56303, server: quickstart.cloudera/10.0.2.15:2181
2016-11-22 11:54:00,814 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server quickstart.cloudera/10.0.2.15:2181, sessionid = 0x1588d7a148b0083, negotiated timeout = 30000
2016-11-22 11:54:00,818 INFO org.apache.solr.common.cloud.ConnectionManager: Watcher org.apache.solr.common.cloud.ConnectionManager@497fec5 name:ZooKeeperConnection Watcher:quickstart.cloudera:2181/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
2016-11-22 11:54:00,819 INFO org.apache.solr.common.cloud.ConnectionManager: Client is connected to ZooKeeper
2016-11-22 11:54:00,819 INFO org.apache.solr.common.cloud.SolrZkClient: Using ZkACLProvider: org.apache.solr.common.cloud.ConfigAwareSaslZkACLProvider
2016-11-22 11:54:00,839 INFO org.apache.solr.common.cloud.ConnectionManager: Watcher org.apache.solr.common.cloud.ConnectionManager@497fec5 name:ZooKeeperConnection Watcher:quickstart.cloudera:2181/solr got event WatchedEvent state:SaslAuthenticated type:None path:null path:null type:None
2016-11-22 11:54:00,993 INFO org.apache.solr.core.CoresLocator: Config-defined core root directory: /var/lib/solr
2016-11-22 11:54:00,996 INFO org.apache.zookeeper.ZooKeeper: Session: 0x1588d7a148b0083 closed
2016-11-22 11:54:00,996 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2016-11-22 11:54:01,000 INFO org.apache.solr.common.cloud.SolrZkClient: Using ZkACLProvider: org.apache.solr.common.cloud.ConfigAwareSaslZkACLProvider
2016-11-22 11:54:01,001 INFO org.apache.solr.common.cloud.SolrZkClient: Using ZkACLProvider: org.apache.solr.common.cloud.ConfigAwareSaslZkACLProvider
2016-11-22 11:54:01,001 INFO org.apache.solr.common.cloud.SolrZkClient: Using default ZkCredentialsProvider
2016-11-22 11:54:01,001 INFO org.apache.solr.servlet.SolrHadoopAuthenticationFilter: Connecting to ZooKeeper with SASL/Kerberos
2016-11-22 11:54:01,027 INFO org.apache.solr.client.solrj.impl.HttpClientUtil: Setting up SPNego auth with config: jaas.conf
2016-11-22 11:54:01,063 WARN org.apache.curator.CuratorZookeeperClient: session timeout [15000] is less than connection timeout [30000]
2016-11-22 11:54:01,112 INFO org.apache.curator.framework.imps.CuratorFrameworkImpl: Starting
2016-11-22 11:54:01,117 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=quickstart.cloudera:2181 sessionTimeout=15000 watcher=org.apache.curator.ConnectionState@775dfcbc
2016-11-22 11:54:01,122 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2016-11-22 11:54:01,125 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server quickstart.cloudera/10.0.2.15:2181. Will attempt to SASL-authenticate using Login Context section 'Client'
2016-11-22 11:54:01,128 INFO org.apache.zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.0.2.15:56304, server: quickstart.cloudera/10.0.2.15:2181
2016-11-22 11:54:01,130 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server quickstart.cloudera/10.0.2.15:2181, sessionid = 0x1588d7a148b0084, negotiated timeout = 15000
2016-11-22 11:54:01,148 INFO org.apache.curator.framework.state.ConnectionStateManager: State change: CONNECTED
2016-11-22 11:54:01,185 INFO org.apache.hadoop.security.authentication.util.ZKSignerSecretProvider: The secret znode already exists, retrieving data
2016-11-22 11:54:01,205 INFO org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler: Login using keytab solr.keytab, for principal HTTP/quickstart.cloudera@QUICKSTART.CLOUDERA
2016-11-22 11:54:01,302 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
2016-11-22 11:54:01,341 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Starting expired delegation token remover thread, tokenRemoverScanInterval=60 min(s)
2016-11-22 11:54:01,359 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
2016-11-22 11:54:01,376 INFO org.apache.solr.servlet.SolrDispatchFilter: SolrDispatchFilter.init()
2016-11-22 11:54:01,376 INFO org.apache.solr.core.SolrResourceLoader: No /solr/home in JNDI
2016-11-22 11:54:01,376 INFO org.apache.solr.core.SolrResourceLoader: using system property solr.solr.home: /var/lib/solr
2016-11-22 11:54:01,376 INFO org.apache.solr.core.SolrResourceLoader: new SolrResourceLoader for directory: '/var/lib/solr/'
2016-11-22 11:54:01,389 INFO org.apache.solr.servlet.SolrDispatchFilter: Trying to read solr.xml from quickstart.cloudera:2181/solr
2016-11-22 11:54:01,390 INFO org.apache.solr.common.cloud.SolrZkClient: Using default ZkCredentialsProvider
2016-11-22 11:54:01,390 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=quickstart.cloudera:2181/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@325b77ed
2016-11-22 11:54:01,390 INFO org.apache.solr.common.cloud.ConnectionManager: Waiting for client to connect to ZooKeeper
2016-11-22 11:54:01,391 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2016-11-22 11:54:01,392 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server quickstart.cloudera/10.0.2.15:2181. Will attempt to SASL-authenticate using Login Context section 'Client'
2016-11-22 11:54:01,393 INFO org.apache.zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.0.2.15:56305, server: quickstart.cloudera/10.0.2.15:2181
2016-11-22 11:54:01,399 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server quickstart.cloudera/10.0.2.15:2181, sessionid = 0x1588d7a148b0085, negotiated timeout = 30000
2016-11-22 11:54:01,400 INFO org.apache.solr.common.cloud.ConnectionManager: Watcher org.apache.solr.common.cloud.ConnectionManager@325b77ed name:ZooKeeperConnection Watcher:quickstart.cloudera:2181/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
2016-11-22 11:54:01,400 INFO org.apache.solr.common.cloud.ConnectionManager: Client is connected to ZooKeeper
2016-11-22 11:54:01,400 INFO org.apache.solr.common.cloud.SolrZkClient: Using ZkACLProvider: org.apache.solr.common.cloud.ConfigAwareSaslZkACLProvider
2016-11-22 11:54:01,411 INFO org.apache.solr.common.cloud.ConnectionManager: Watcher org.apache.solr.common.cloud.ConnectionManager@325b77ed name:ZooKeeperConnection Watcher:quickstart.cloudera:2181/solr got event WatchedEvent state:SaslAuthenticated type:None path:null path:null type:None
2016-11-22 11:54:01,447 INFO org.apache.solr.core.CoresLocator: Config-defined core root directory: /var/lib/solr
2016-11-22 11:54:01,448 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2016-11-22 11:54:01,449 INFO org.apache.zookeeper.ZooKeeper: Session: 0x1588d7a148b0085 closed
2016-11-22 11:54:01,456 INFO org.apache.solr.core.CoreContainer: New CoreContainer 932454647
2016-11-22 11:54:01,456 INFO org.apache.solr.core.CoreContainer: Loading cores into CoreContainer [instanceDir=/var/lib/solr/]
2016-11-22 11:54:01,471 INFO org.apache.solr.handler.component.HttpShardHandlerFactory: Setting socketTimeout to: 0
2016-11-22 11:54:01,472 INFO org.apache.solr.handler.component.HttpShardHandlerFactory: Setting urlScheme to: null
2016-11-22 11:54:01,472 INFO org.apache.solr.handler.component.HttpShardHandlerFactory: Setting connTimeout to: 0
2016-11-22 11:54:01,472 INFO org.apache.solr.handler.component.HttpShardHandlerFactory: Setting maxConnectionsPerHost to: 20
2016-11-22 11:54:01,472 INFO org.apache.solr.handler.component.HttpShardHandlerFactory: Setting corePoolSize to: 0
2016-11-22 11:54:01,472 INFO org.apache.solr.handler.component.HttpShardHandlerFactory: Setting maximumPoolSize to: 2147483647
2016-11-22 11:54:01,472 INFO org.apache.solr.handler.component.HttpShardHandlerFactory: Setting maxThreadIdleTime to: 5
2016-11-22 11:54:01,472 INFO org.apache.solr.handler.component.HttpShardHandlerFactory: Setting sizeOfQueue to: -1
2016-11-22 11:54:01,472 INFO org.apache.solr.handler.component.HttpShardHandlerFactory: Setting fairnessPolicy to: false
2016-11-22 11:54:01,472 INFO org.apache.solr.handler.component.HttpShardHandlerFactory: Setting useRetries to: false
2016-11-22 11:54:01,577 INFO org.apache.solr.client.solrj.impl.HttpClientUtil: Setting up SPNego auth with config: jaas.conf
2016-11-22 11:54:01,585 INFO org.apache.solr.client.solrj.impl.HttpClientUtil: Setting up SPNego auth with config: jaas.conf
2016-11-22 11:54:01,587 INFO org.apache.solr.logging.LogWatcher: SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
2016-11-22 11:54:01,588 INFO org.apache.solr.logging.LogWatcher: Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
2016-11-22 11:54:01,589 INFO org.apache.solr.core.CoreContainer: Host Name: quickstart.cloudera
2016-11-22 11:54:01,589 INFO org.apache.solr.core.ZkContainer: Zookeeper client=quickstart.cloudera:2181/solr
2016-11-22 11:54:01,598 INFO org.apache.solr.cloud.ZkController: zkHost includes chroot
2016-11-22 11:54:01,598 INFO org.apache.solr.common.cloud.SolrZkClient: Using default ZkCredentialsProvider
2016-11-22 11:54:01,598 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=quickstart.cloudera:2181 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@6bf1cbc7
2016-11-22 11:54:01,599 INFO org.apache.solr.common.cloud.ConnectionManager: Waiting for client to connect to ZooKeeper
2016-11-22 11:54:01,600 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2016-11-22 11:54:01,600 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server quickstart.cloudera/10.0.2.15:2181. Will attempt to SASL-authenticate using Login Context section 'Client'
2016-11-22 11:54:01,602 INFO org.apache.zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.0.2.15:56306, server: quickstart.cloudera/10.0.2.15:2181
2016-11-22 11:54:01,604 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server quickstart.cloudera/10.0.2.15:2181, sessionid = 0x1588d7a148b0086, negotiated timeout = 60000
2016-11-22 11:54:01,605 INFO org.apache.solr.common.cloud.ConnectionManager: Watcher org.apache.solr.common.cloud.ConnectionManager@6bf1cbc7 name:ZooKeeperConnection Watcher:quickstart.cloudera:2181 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
2016-11-22 11:54:01,605 INFO org.apache.solr.common.cloud.ConnectionManager: Client is connected to ZooKeeper
2016-11-22 11:54:01,605 INFO org.apache.solr.common.cloud.SolrZkClient: Using ZkACLProvider: org.apache.solr.common.cloud.ConfigAwareSaslZkACLProvider
2016-11-22 11:54:01,624 INFO org.apache.solr.common.cloud.ConnectionManager: Watcher org.apache.solr.common.cloud.ConnectionManager@6bf1cbc7 name:ZooKeeperConnection Watcher:quickstart.cloudera:2181 got event WatchedEvent state:SaslAuthenticated type:None path:null path:null type:None
2016-11-22 11:54:01,626 INFO org.apache.zookeeper.ZooKeeper: Session: 0x1588d7a148b0086 closed
2016-11-22 11:54:01,626 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2016-11-22 11:54:01,630 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=quickstart.cloudera:2181/solr sessionTimeout=15000 watcher=org.apache.solr.common.cloud.ConnectionManager@5fc3184b
2016-11-22 11:54:01,631 INFO org.apache.solr.common.cloud.ConnectionManager: Waiting for client to connect to ZooKeeper
2016-11-22 11:54:01,631 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2016-11-22 11:54:01,632 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server quickstart.cloudera/10.0.2.15:2181. Will attempt to SASL-authenticate using Login Context section 'Client'
2016-11-22 11:54:01,632 INFO org.apache.zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.0.2.15:56307, server: quickstart.cloudera/10.0.2.15:2181
2016-11-22 11:54:01,635 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server quickstart.cloudera/10.0.2.15:2181, sessionid = 0x1588d7a148b0087, negotiated timeout = 15000
2016-11-22 11:54:01,635 INFO org.apache.solr.common.cloud.ConnectionManager: Watcher org.apache.solr.common.cloud.ConnectionManager@5fc3184b name:ZooKeeperConnection Watcher:quickstart.cloudera:2181/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
2016-11-22 11:54:01,635 INFO org.apache.solr.common.cloud.ConnectionManager: Client is connected to ZooKeeper
2016-11-22 11:54:01,648 INFO org.apache.solr.common.cloud.ConnectionManager: Watcher org.apache.solr.common.cloud.ConnectionManager@5fc3184b name:ZooKeeperConnection Watcher:quickstart.cloudera:2181/solr got event WatchedEvent state:SaslAuthenticated type:None path:null path:null type:None
2016-11-22 11:54:01,677 INFO org.apache.solr.cloud.ZkController: Register node as live in ZooKeeper:/live_nodes/quickstart.cloudera:8983_solr
2016-11-22 11:54:01,681 INFO org.apache.solr.common.cloud.SolrZkClient: makePath: /live_nodes/quickstart.cloudera:8983_solr
2016-11-22 11:54:01,692 INFO org.apache.solr.cloud.Overseer: Overseer (id=null) closing
2016-11-22 11:54:01,696 INFO org.apache.solr.cloud.ElectionContext: I am going to be the leader quickstart.cloudera:8983_solr
2016-11-22 11:54:01,698 INFO org.apache.solr.common.cloud.SolrZkClient: makePath: /overseer_elect/leader
2016-11-22 11:54:01,705 INFO org.apache.solr.cloud.Overseer: Overseer (id=96982947458646151-quickstart.cloudera:8983_solr-n_0000000034) starting
2016-11-22 11:54:01,759 INFO org.apache.solr.cloud.OverseerAutoReplicaFailoverThread: Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
2016-11-22 11:54:01,783 INFO org.apache.solr.cloud.OverseerTaskProcessor: Process current queue of overseer operations
2016-11-22 11:54:01,786 INFO org.apache.solr.common.cloud.ZkStateReader: Updating cluster state from ZooKeeper... 
2016-11-22 11:54:01,807 INFO org.apache.solr.cloud.Overseer: Starting to work on the main queue
2016-11-22 11:54:01,814 INFO org.apache.solr.cloud.Overseer: DownNode state invoked for node: quickstart.cloudera:8983_solr
2016-11-22 11:54:01,828 INFO org.apache.solr.cloud.DistributedQueue: NodeChildrenChanged fired on path /overseer/queue state SyncConnected
2016-11-22 11:54:01,839 INFO org.apache.solr.core.CoresLocator: Looking for core definitions underneath /var/lib/solr
2016-11-22 11:54:01,840 INFO org.apache.solr.common.cloud.ZkStateReader: A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
2016-11-22 11:54:01,849 INFO org.apache.solr.core.CoresLocator: Found 0 core definitions
2016-11-22 11:54:01,852 INFO org.apache.solr.servlet.SolrDispatchFilter: user.dir=/var/run/cloudera-scm-agent/process/373-solr-SOLR_SERVER
2016-11-22 11:54:01,852 INFO org.apache.solr.servlet.SolrDispatchFilter: SolrDispatchFilter.init() done
2016-11-22 11:54:38,306 INFO org.apache.solr.servlet.SolrDispatchFilter: [admin] webapp=null path=/admin/cores params={action=STATUS&wt=json} status=0 QTime=57

 

With Sentry Enabled:

2016-11-22 11:56:02,024 INFO org.apache.solr.core.SolrResourceLoader: No /solr/home in JNDI
2016-11-22 11:56:02,025 INFO org.apache.solr.core.SolrResourceLoader: using system property solr.solr.home: /var/lib/solr
2016-11-22 11:56:02,027 INFO org.apache.solr.core.SolrResourceLoader: new SolrResourceLoader for directory: '/var/lib/solr/'
2016-11-22 11:56:02,245 INFO org.apache.solr.servlet.SolrDispatchFilter: Trying to read solr.xml from quickstart.cloudera:2181/solr
2016-11-22 11:56:02,259 INFO org.apache.solr.common.cloud.SolrZkClient: Using default ZkCredentialsProvider
2016-11-22 11:56:02,273 INFO org.apache.zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.5-cdh5.8.0--1, built on 06/16/2016 19:37 GMT
2016-11-22 11:56:02,273 INFO org.apache.zookeeper.ZooKeeper: Client environment:host.name=quickstart.cloudera
2016-11-22 11:56:02,273 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.version=1.7.0_67
2016-11-22 11:56:02,273 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
2016-11-22 11:56:02,273 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.home=/usr/java/jdk1.7.0_67-cloudera/jre
2016-11-22 11:56:02,274 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.class.path=/usr/lib/bigtop-tomcat/bin/bootstrap.jar
2016-11-22 11:56:02,274 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2016-11-22 11:56:02,274 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/var/lib/solr/
2016-11-22 11:56:02,274 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
2016-11-22 11:56:02,274 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.name=Linux
2016-11-22 11:56:02,274 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.arch=amd64
2016-11-22 11:56:02,274 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.version=2.6.32-573.el6.x86_64
2016-11-22 11:56:02,274 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.name=solr
2016-11-22 11:56:02,274 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.home=/var/lib/solr
2016-11-22 11:56:02,274 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.dir=/var/run/cloudera-scm-agent/process/375-solr-SOLR_SERVER
2016-11-22 11:56:02,275 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=quickstart.cloudera:2181/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@64ee793
2016-11-22 11:56:02,293 INFO org.apache.solr.common.cloud.ConnectionManager: Waiting for client to connect to ZooKeeper
2016-11-22 11:56:02,441 INFO org.apache.zookeeper.Login: successfully logged in.
2016-11-22 11:56:02,443 INFO org.apache.zookeeper.Login: TGT refresh thread started.
2016-11-22 11:56:02,448 INFO org.apache.zookeeper.Login: TGT valid starting at:        Tue Nov 22 11:56:02 PST 2016
2016-11-22 11:56:02,448 INFO org.apache.zookeeper.Login: TGT expires:                  Wed Nov 23 11:56:02 PST 2016
2016-11-22 11:56:02,449 INFO org.apache.zookeeper.Login: TGT refresh sleeping until: Wed Nov 23 07:28:24 PST 2016
2016-11-22 11:56:02,449 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2016-11-22 11:56:02,456 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server quickstart.cloudera/10.0.2.15:2181. Will attempt to SASL-authenticate using Login Context section 'Client'
2016-11-22 11:56:02,460 INFO org.apache.zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.0.2.15:56561, server: quickstart.cloudera/10.0.2.15:2181
2016-11-22 11:56:02,470 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server quickstart.cloudera/10.0.2.15:2181, sessionid = 0x1588d7a148b008d, negotiated timeout = 30000
2016-11-22 11:56:02,476 INFO org.apache.solr.common.cloud.ConnectionManager: Watcher org.apache.solr.common.cloud.ConnectionManager@64ee793 name:ZooKeeperConnection Watcher:quickstart.cloudera:2181/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
2016-11-22 11:56:02,477 INFO org.apache.solr.common.cloud.ConnectionManager: Client is connected to ZooKeeper
2016-11-22 11:56:02,477 INFO org.apache.solr.common.cloud.SolrZkClient: Using ZkACLProvider: org.apache.solr.common.cloud.ConfigAwareSaslZkACLProvider
2016-11-22 11:56:02,540 INFO org.apache.solr.common.cloud.ConnectionManager: Watcher org.apache.solr.common.cloud.ConnectionManager@64ee793 name:ZooKeeperConnection Watcher:quickstart.cloudera:2181/solr got event WatchedEvent state:SaslAuthenticated type:None path:null path:null type:None
2016-11-22 11:56:02,638 INFO org.apache.solr.core.CoresLocator: Config-defined core root directory: /var/lib/solr
2016-11-22 11:56:02,641 INFO org.apache.zookeeper.ZooKeeper: Session: 0x1588d7a148b008d closed
2016-11-22 11:56:02,641 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2016-11-22 11:56:02,645 INFO org.apache.solr.common.cloud.SolrZkClient: Using ZkACLProvider: org.apache.solr.common.cloud.ConfigAwareSaslZkACLProvider
2016-11-22 11:56:02,645 INFO org.apache.solr.common.cloud.SolrZkClient: Using ZkACLProvider: org.apache.solr.common.cloud.ConfigAwareSaslZkACLProvider
2016-11-22 11:56:02,646 INFO org.apache.solr.common.cloud.SolrZkClient: Using default ZkCredentialsProvider
2016-11-22 11:56:02,646 INFO org.apache.solr.servlet.SolrHadoopAuthenticationFilter: Connecting to ZooKeeper with SASL/Kerberos
2016-11-22 11:56:02,680 INFO org.apache.solr.client.solrj.impl.HttpClientUtil: Setting up SPNego auth with config: jaas.conf
2016-11-22 11:56:02,709 WARN org.apache.curator.CuratorZookeeperClient: session timeout [15000] is less than connection timeout [30000]
2016-11-22 11:56:02,751 INFO org.apache.curator.framework.imps.CuratorFrameworkImpl: Starting
2016-11-22 11:56:02,754 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=quickstart.cloudera:2181 sessionTimeout=15000 watcher=org.apache.curator.ConnectionState@6d348a50
2016-11-22 11:56:02,755 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2016-11-22 11:56:02,759 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server quickstart.cloudera/10.0.2.15:2181. Will attempt to SASL-authenticate using Login Context section 'Client'
2016-11-22 11:56:02,759 INFO org.apache.zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.0.2.15:56563, server: quickstart.cloudera/10.0.2.15:2181
2016-11-22 11:56:02,764 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server quickstart.cloudera/10.0.2.15:2181, sessionid = 0x1588d7a148b008e, negotiated timeout = 15000
2016-11-22 11:56:02,773 INFO org.apache.curator.framework.state.ConnectionStateManager: State change: CONNECTED
2016-11-22 11:56:02,800 INFO org.apache.hadoop.security.authentication.util.ZKSignerSecretProvider: The secret znode already exists, retrieving data
2016-11-22 11:56:02,816 INFO org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler: Login using keytab solr.keytab, for principal HTTP/quickstart.cloudera@QUICKSTART.CLOUDERA
2016-11-22 11:56:02,926 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
2016-11-22 11:56:02,957 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Starting expired delegation token remover thread, tokenRemoverScanInterval=60 min(s)
2016-11-22 11:56:02,971 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
2016-11-22 11:56:02,990 INFO org.apache.solr.servlet.SolrDispatchFilter: SolrDispatchFilter.init()
2016-11-22 11:56:02,991 INFO org.apache.solr.core.SolrResourceLoader: No /solr/home in JNDI
2016-11-22 11:56:02,991 INFO org.apache.solr.core.SolrResourceLoader: using system property solr.solr.home: /var/lib/solr
2016-11-22 11:56:02,991 INFO org.apache.solr.core.SolrResourceLoader: new SolrResourceLoader for directory: '/var/lib/solr/'
2016-11-22 11:56:03,006 INFO org.apache.solr.servlet.SolrDispatchFilter: Trying to read solr.xml from quickstart.cloudera:2181/solr
2016-11-22 11:56:03,007 INFO org.apache.solr.common.cloud.SolrZkClient: Using default ZkCredentialsProvider
2016-11-22 11:56:03,007 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=quickstart.cloudera:2181/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@21b21f5c
2016-11-22 11:56:03,007 INFO org.apache.solr.common.cloud.ConnectionManager: Waiting for client to connect to ZooKeeper
2016-11-22 11:56:03,009 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2016-11-22 11:56:03,010 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server quickstart.cloudera/10.0.2.15:2181. Will attempt to SASL-authenticate using Login Context section 'Client'
2016-11-22 11:56:03,010 INFO org.apache.zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.0.2.15:56564, server: quickstart.cloudera/10.0.2.15:2181
2016-11-22 11:56:03,013 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server quickstart.cloudera/10.0.2.15:2181, sessionid = 0x1588d7a148b008f, negotiated timeout = 30000
2016-11-22 11:56:03,013 INFO org.apache.solr.common.cloud.ConnectionManager: Watcher org.apache.solr.common.cloud.ConnectionManager@21b21f5c name:ZooKeeperConnection Watcher:quickstart.cloudera:2181/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
2016-11-22 11:56:03,013 INFO org.apache.solr.common.cloud.ConnectionManager: Client is connected to ZooKeeper
2016-11-22 11:56:03,013 INFO org.apache.solr.common.cloud.SolrZkClient: Using ZkACLProvider: org.apache.solr.common.cloud.ConfigAwareSaslZkACLProvider
2016-11-22 11:56:03,031 INFO org.apache.solr.common.cloud.ConnectionManager: Watcher org.apache.solr.common.cloud.ConnectionManager@21b21f5c name:ZooKeeperConnection Watcher:quickstart.cloudera:2181/solr got event WatchedEvent state:SaslAuthenticated type:None path:null path:null type:None
2016-11-22 11:56:03,066 INFO org.apache.solr.core.CoresLocator: Config-defined core root directory: /var/lib/solr
2016-11-22 11:56:03,068 INFO org.apache.zookeeper.ZooKeeper: Session: 0x1588d7a148b008f closed
2016-11-22 11:56:03,068 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2016-11-22 11:56:03,074 INFO org.apache.solr.core.CoreContainer: New CoreContainer 11873731
2016-11-22 11:56:03,074 INFO org.apache.solr.core.CoreContainer: Loading cores into CoreContainer [instanceDir=/var/lib/solr/]
2016-11-22 11:56:03,092 INFO org.apache.solr.handler.component.HttpShardHandlerFactory: Setting socketTimeout to: 0
2016-11-22 11:56:03,092 INFO org.apache.solr.handler.component.HttpShardHandlerFactory: Setting urlScheme to: null
2016-11-22 11:56:03,092 INFO org.apache.solr.handler.component.HttpShardHandlerFactory: Setting connTimeout to: 0
2016-11-22 11:56:03,092 INFO org.apache.solr.handler.component.HttpShardHandlerFactory: Setting maxConnectionsPerHost to: 20
2016-11-22 11:56:03,092 INFO org.apache.solr.handler.component.HttpShardHandlerFactory: Setting corePoolSize to: 0
2016-11-22 11:56:03,092 INFO org.apache.solr.handler.component.HttpShardHandlerFactory: Setting maximumPoolSize to: 2147483647
2016-11-22 11:56:03,092 INFO org.apache.solr.handler.component.HttpShardHandlerFactory: Setting maxThreadIdleTime to: 5
2016-11-22 11:56:03,092 INFO org.apache.solr.handler.component.HttpShardHandlerFactory: Setting sizeOfQueue to: -1
2016-11-22 11:56:03,092 INFO org.apache.solr.handler.component.HttpShardHandlerFactory: Setting fairnessPolicy to: false
2016-11-22 11:56:03,092 INFO org.apache.solr.handler.component.HttpShardHandlerFactory: Setting useRetries to: false
2016-11-22 11:56:03,191 INFO org.apache.solr.client.solrj.impl.HttpClientUtil: Setting up SPNego auth with config: jaas.conf
2016-11-22 11:56:03,198 INFO org.apache.solr.client.solrj.impl.HttpClientUtil: Setting up SPNego auth with config: jaas.conf
2016-11-22 11:56:03,201 INFO org.apache.solr.logging.LogWatcher: SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
2016-11-22 11:56:03,202 INFO org.apache.solr.logging.LogWatcher: Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
2016-11-22 11:56:03,203 INFO org.apache.solr.core.CoreContainer: Host Name: quickstart.cloudera
2016-11-22 11:56:03,204 INFO org.apache.solr.core.ZkContainer: Zookeeper client=quickstart.cloudera:2181/solr
2016-11-22 11:56:03,211 INFO org.apache.solr.cloud.ZkController: zkHost includes chroot
2016-11-22 11:56:03,212 INFO org.apache.solr.common.cloud.SolrZkClient: Using default ZkCredentialsProvider
2016-11-22 11:56:03,212 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=quickstart.cloudera:2181 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@5d03575d
2016-11-22 11:56:03,212 INFO org.apache.solr.common.cloud.ConnectionManager: Waiting for client to connect to ZooKeeper
2016-11-22 11:56:03,213 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2016-11-22 11:56:03,214 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server quickstart.cloudera/10.0.2.15:2181. Will attempt to SASL-authenticate using Login Context section 'Client'
2016-11-22 11:56:03,214 INFO org.apache.zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.0.2.15:56565, server: quickstart.cloudera/10.0.2.15:2181
2016-11-22 11:56:03,217 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server quickstart.cloudera/10.0.2.15:2181, sessionid = 0x1588d7a148b0090, negotiated timeout = 60000
2016-11-22 11:56:03,217 INFO org.apache.solr.common.cloud.ConnectionManager: Watcher org.apache.solr.common.cloud.ConnectionManager@5d03575d name:ZooKeeperConnection Watcher:quickstart.cloudera:2181 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
2016-11-22 11:56:03,217 INFO org.apache.solr.common.cloud.ConnectionManager: Client is connected to ZooKeeper
2016-11-22 11:56:03,217 INFO org.apache.solr.common.cloud.SolrZkClient: Using ZkACLProvider: org.apache.solr.common.cloud.ConfigAwareSaslZkACLProvider
2016-11-22 11:56:03,227 INFO org.apache.solr.common.cloud.ConnectionManager: Watcher org.apache.solr.common.cloud.ConnectionManager@5d03575d name:ZooKeeperConnection Watcher:quickstart.cloudera:2181 got event WatchedEvent state:SaslAuthenticated type:None path:null path:null type:None
2016-11-22 11:56:03,229 INFO org.apache.zookeeper.ZooKeeper: Session: 0x1588d7a148b0090 closed
2016-11-22 11:56:03,229 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2016-11-22 11:56:03,232 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=quickstart.cloudera:2181/solr sessionTimeout=15000 watcher=org.apache.solr.common.cloud.ConnectionManager@440e66e8
2016-11-22 11:56:03,233 INFO org.apache.solr.common.cloud.ConnectionManager: Waiting for client to connect to ZooKeeper
2016-11-22 11:56:03,233 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2016-11-22 11:56:03,234 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server quickstart.cloudera/10.0.2.15:2181. Will attempt to SASL-authenticate using Login Context section 'Client'
2016-11-22 11:56:03,234 INFO org.apache.zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.0.2.15:56566, server: quickstart.cloudera/10.0.2.15:2181
2016-11-22 11:56:03,237 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server quickstart.cloudera/10.0.2.15:2181, sessionid = 0x1588d7a148b0091, negotiated timeout = 15000
2016-11-22 11:56:03,238 INFO org.apache.solr.common.cloud.ConnectionManager: Watcher org.apache.solr.common.cloud.ConnectionManager@440e66e8 name:ZooKeeperConnection Watcher:quickstart.cloudera:2181/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
2016-11-22 11:56:03,238 INFO org.apache.solr.common.cloud.ConnectionManager: Client is connected to ZooKeeper
2016-11-22 11:56:03,247 INFO org.apache.solr.common.cloud.ConnectionManager: Watcher org.apache.solr.common.cloud.ConnectionManager@440e66e8 name:ZooKeeperConnection Watcher:quickstart.cloudera:2181/solr got event WatchedEvent state:SaslAuthenticated type:None path:null path:null type:None
2016-11-22 11:56:03,281 INFO org.apache.solr.cloud.ZkController: Register node as live in ZooKeeper:/live_nodes/quickstart.cloudera:8983_solr
2016-11-22 11:56:03,283 INFO org.apache.solr.common.cloud.SolrZkClient: makePath: /live_nodes/quickstart.cloudera:8983_solr
2016-11-22 11:56:03,294 INFO org.apache.solr.cloud.Overseer: Overseer (id=null) closing
2016-11-22 11:56:03,298 INFO org.apache.solr.cloud.ElectionContext: I am going to be the leader quickstart.cloudera:8983_solr
2016-11-22 11:56:03,300 INFO org.apache.solr.common.cloud.SolrZkClient: makePath: /overseer_elect/leader
2016-11-22 11:56:03,304 INFO org.apache.solr.cloud.Overseer: Overseer (id=96982947458646161-quickstart.cloudera:8983_solr-n_0000000035) starting
2016-11-22 11:56:03,358 INFO org.apache.solr.cloud.OverseerAutoReplicaFailoverThread: Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
2016-11-22 11:56:03,359 INFO org.apache.solr.cloud.OverseerTaskProcessor: Process current queue of overseer operations
2016-11-22 11:56:03,361 INFO org.apache.solr.common.cloud.ZkStateReader: Updating cluster state from ZooKeeper... 
2016-11-22 11:56:03,375 INFO org.apache.solr.cloud.Overseer: Starting to work on the main queue
2016-11-22 11:56:03,381 ERROR org.apache.solr.servlet.SolrDispatchFilter: Could not start Solr. Check solr/home property and the logs
2016-11-22 11:56:03,405 INFO org.apache.solr.cloud.Overseer: DownNode state invoked for node: quickstart.cloudera:8983_solr
2016-11-22 11:56:03,413 INFO org.apache.solr.cloud.DistributedQueue: NodeChildrenChanged fired on path /overseer/queue state SyncConnected
2016-11-22 11:56:03,415 INFO org.apache.solr.common.cloud.ZkStateReader: A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
2016-11-22 11:56:03,458 ERROR org.apache.solr.core.SolrCore: null:java.lang.NoSuchFieldError: LISTSNAPSHOTS
	at org.apache.solr.handler.admin.SecureCollectionsHandler.<clinit>(SecureCollectionsHandler.java:38)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:270)
	at org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:474)
	at org.apache.solr.core.SolrResourceLoader.newInstance(SolrResourceLoader.java:593)
	at org.apache.solr.core.CoreContainer.createHandler(CoreContainer.java:784)
	at org.apache.solr.core.CoreContainer.load(CoreContainer.java:230)
	at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:237)
	at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:182)
	at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:298)
	at org.apache.catalina.core.ApplicationFilterConfig.<init>(ApplicationFilterConfig.java:119)
	at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4076)
	at org.apache.catalina.core.StandardContext.start(StandardContext.java:4730)
	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:802)
	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:583)
	at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1080)
	at org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:1003)
	at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:507)
	at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1322)
	at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:325)
	at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
	at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1068)
	at org.apache.catalina.core.StandardHost.start(StandardHost.java:822)
	at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1060)
	at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
	at org.apache.catalina.core.StandardService.start(StandardService.java:525)
	at org.apache.catalina.core.StandardServer.start(StandardServer.java:759)
	at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
	at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)

2016-11-22 11:56:03,467 INFO org.apache.zookeeper.ZooKeeper: Session: 0x1588d7a148b008e closed
2016-11-22 11:56:03,467 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2016-11-22 11:56:03,467 ERROR org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted

 

Thanks in advance for any help!

Jared-

 

 

 

2,973
0 Kudos
1
Who agreed with this topic