After recently upgrading SmartSense, you successfully log into Activity-Explorer and attempt to execute any pre-installed notebooks only to encounter one of the following errors:
RpcRetryingCaller {globalStartTime=1473694857754, pause=100, retries=35},
org.apache.hadoop.hbase.MasterNotRunningException:
com.google.protobuf.ServiceException:
org.apache.hadoop.hbase.exceptions.ConnectionClosingException:
Call to node1.abcxyz.com/192.168.1.1:61300 failed on local exception:
org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to node1.abcxyz.com/192.168.1.1:61300 is closing. Call id=4, waitTime=2
or
192.168.1.1:61300 is closing. Call id=32, waitTime=4
RpcRetryingCaller{globalStartTime=1473456721387, pause=100, retries=35},
org.apache.hadoop.hbase.MasterNotRunningException:
com.google.protobuf.ServiceException:
java.io.IOException: Broken pipe
RpcRetryingCaller{globalStartTime=1473456721387, pause=100, retries=35},
org.apache.hadoop.hbase.MasterNotRunningException:
com.google.protobuf.ServiceException:
org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to node1.abcxyz.org/192.168.1.1:61300 failed on local exception:
org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to node1.abcxyz.org/192.168.1.1:61300 is closing
Diagnosis:
This can occur when certain dependent libraries were not correctly loaded after the upgrade of SmartSense.
Solution:
Try the following steps to remedy the solution:
Restart AMS (specifically looking for the collector to restart) then restart Activity-Analyzer and last restart Activity-Explorer. These restarts should correct the loading issue of the dependent libraries.
If the problem still persists, add the following export statement to the zeppelin-env.sh (typically found at /usr/hdp/share/hst/activity-explorer/conf) on the node where Activity-Explorer is installed - once added restart Activity-Explorer: