master2.corp.mirrorplus.com,16000 is the active master, not the host whose logs you pasted. ... View more

That's a latency of 55hrs or 2.3days for a read. That seems extremely unlikely that HBase would even be functioning if that's the case as the default RPC timeout for HBase is on the order of a few minutes. It seems like that graph may be inaccurate. ... View more

Yes, seeing "[Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]" errors in a sevice after it has been running for some time implies that the issue is ACCUMULO-4069 ... View more

Great, that's a very helpful message. The Accumulo Master asked this tabletserver to stop. This happens for one of two reasons: 1. The "hold time" for this server (the amount of time that mutations are being held because of a flush/minor-compaction that is in progress) exceeds a given threshold. By default, this value is 5minutes and is defined by tserver.hold.time.max in accumulo-site.xml 2. The Master periodically asks every tabletserver for a status report. If the Master fails to receive a status report from a TabletServer 3 times in a row, it will request that it shuts down (as it implies that the tabletserver is in a bad state). Both of these cases will result in a log message written to the Master log file. Please check the Master log file shortly before 2016-03-09 20:35:36,971 to understand which reason it was. ... View more

@AR Like I said in my other comment, this exception does not cause the tabletserver to fail. Please collect all logs and .out/.err for the tabletserver after a failure *but before* restarting the process. ... View more

https://issues.apache.org/jira/browse/ACCUMULO-4059 ... View more

Login failure for 2181 from keytab /hbase Is very strange looking ("2181" should be the user and "/hbase" should be the keytab). Maybe some of the configuration values you're writing are incorrect? If you take a look at https://community.hortonworks.com/articles/808/phoenix-jdbc-client-setup.html, you can see how Kerberos information can be provided via the JDBC URL. It almost seems like the last two elements of your JDBC URL are getting parsed as the principal and keytab -- it should really only look for the 4th and 5th property (not just the last two properties). Maybe there is a bug in Phoenix here? It's a starting point to investigate. However, if your Spark code is only running locally, since you've already logged in (should probably error out it the login fails instead of just printing the exception, though), you shouldn't have to pass this additional Kerberos login information down into Phoenix. ... View more

Also worth mentioning that sqlline.py doesn't require you do run the QueryServer. That's only for sqlline-thin.py (or if you have other reasons you want to run the queryserver). ... View more