Symptoms: After recently upgrading SmartSense, you successfully log into Activity-Explorer and attempt to execute any pre-installed notebooks only to encounter one of the following errors: RpcRetryingCaller {globalStartTime=1473694857754, pause=100, retries=35}, org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to node1.abcxyz.com/192.168.1.1:61300 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to node1.abcxyz.com/192.168.1.1:61300 is closing. Call id=4, waitTime=2 or 192.168.1.1:61300 is closing. Call id=32, waitTime=4 RpcRetryingCaller{globalStartTime=1473456721387, pause=100, retries=35}, org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: java.io.IOException: Broken pipe RpcRetryingCaller{globalStartTime=1473456721387, pause=100, retries=35}, org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to node1.abcxyz.org/192.168.1.1:61300 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to node1.abcxyz.org/192.168.1.1:61300 is closing Diagnosis: This can occur when certain dependent libraries were not correctly loaded after the upgrade of SmartSense. Solution: Try the following steps to remedy the solution: Restart AMS (specifically looking for the collector to restart) then restart Activity-Analyzer and last restart Activity-Explorer. These restarts should correct the loading issue of the dependent libraries. If the problem still persists, add the following export statement to the zeppelin-env.sh (typically found at /usr/hdp/share/hst/activity-explorer/conf) on the node where Activity-Explorer is installed - once added restart Activity-Explorer: export CLASSPATH=${ZEPPELIN_CLASSPATH} ... View more

Symptoms: Bundle capture will fail with the following error found in the hst-agent.log, during the anonymization process: ERROR anonymize.py:62 - Execution of script /usr/jdk64/jdk1.7.0_67/bin/java -Xmx2048m -Xms1024m -Dlog.file.name=anonymization.log -cp :/etc/hst/conf/:/usr/hdp/share/hst/hst-common/lib/* com.hortonworks.smartsense.anonymization.Main -m /var/lib/smartsense/hst-agent/data/tmp/06-01-17.io-a-00000000-c-00000000_hdp_0_2017-06-01_12-00-00 -c /etc/hst/conf/hst-agent.ini failed ERROR anonymize.py:63 - Execution of command returned 1. Exception in thread "main" java.lang.NullPointerException at com.hortonworks.smartsense.anonymization.rules.RuleType.from(RuleType.java:47) at com.hortonworks.smartsense.anonymization.rules.Rules.createRule(Rules.java:161)at com.hortonworks.smartsense.anonymization.rules.Rules.createRules(Rules.java:148)at com.hortonworks.smartsense.anonymization.rules.Rules.createRules(Rules.java:132)at com.hortonworks.smartsense.anonymization.AnonymizerFactory.createAnonymizer(AnonymizerFactory.java:94)at com.hortonworks.smartsense.anonymization.Main.start(Main.java:202) at com.hortonworks.smartsense.anonymization.Main.main(Main.java:294) ERROR AnonymizeBundleCommand.py:60 - Anonymization failed. Please check logs. Traceback (most recent call last): File "/usr/hdp/share/hst/hst-agent/lib/hst_agent/command/AnonymizeBundleCommand.py", line 56, in execute context['bundle_dir'] = anonymizer.anonymize(bundle_dir) File "/usr/hdp/share/hst/hst-agent/lib/hst_agent/anonymize.py", line 64, in anonymize raise Exception("Anonymization failed.") Exception: Anonymization failed. Diagnosis: During the SmartSense HST upgrade process, the anonymization-rules.json file will be modified. This error typically indicates that the process of updating/modifying the anonymization-rules.json file somehow failed to complete successfully. Solution: This exception occurs when the /etc/hst/conf/anonymization-rules.json file does not update as expected during the upgrade process. To correct this you must first replace the anonymization-rules.json with the valid base version and then re-add any customizations that you had prior to the upgrade. Administrators can extract a valid version of the anonymization-rules.json from the RPM using the rpm2cpio utility in Linux or Mac. The extracted file should then be manually copied out to /etc/hst/conf/anonymization-rules.json on the server where the HST server is installed. Once this has been done, you can then re-add any customizations that you had prior to the upgrade via the UI. ... View more

Symptoms: HST Bundle capture fails during the anonymization process with an "Illegal group reference" message exception found in the hst-agent logs: ERROR shell.py:95 - Execution of command returned 1. Exception in thread "main" java.lang.IllegalArgumentException: Illegal group reference         at java.util.regex.Matcher.appendReplacement(Matcher.java:857)         at java.util.regex.Matcher.replaceAll(Matcher.java:955)         at java.lang.String.replaceAll(String.java:2223)         at com.hortonworks.smartsense.anonymization.BundleAnonymizer.handleApplyGroupPatternKeyValue(BundleAnonymizer.java:690)         at com.hortonworks.smartsense.anonymization.BundleAnonymizer.applyGroupPattern(BundleAnonymizer.java:673)         at com.hortonworks.smartsense.anonymization.BundleAnonymizer.applyPropertyRule(BundleAnonymizer.java:612)         at com.hortonworks.smartsense.anonymization.BundleAnonymizer.applyGroupPattern(BundleAnonymizer.java:673)         at com.hortonworks.smartsense.anonymization.BundleAnonymizer.applyPropertyRule(BundleAnonymizer.java:612)         at com.hortonworks.smartsense.anonymization.BundleAnonymizer.applyRules(BundleAnonymizer.java:393)         at com.hortonworks.smartsense.anonymization.BundleAnonymizer.anonymizeFolder(BundleAnonymizer.java:291)         at com.hortonworks.smartsense.anonymization.BundleAnonymizer.anonymizeFolder(BundleAnonymizer.java:259)         at com.hortonworks.smartsense.anonymization.BundleAnonymizer.anonymizeFolder(BundleAnonymizer.java:224)         at com.hortonworks.smartsense.anonymization.Main.run(Main.java:82)         at com.hortonworks.smartsense.anonymization.Main.start(Main.java:210)         at com.hortonworks.smartsense.anonymization.Main.main(Main.java:294) ERROR anonymize.py:67 - Execution of script /usr/lib/jvm/java/bin/java -Xmx2048m -Xms1024m -Dlog.file.name=anonymization.log -Djava.io.tmpdir=/var/lib/smartsense/hst-agent/data/tmp -cp :/etc/hst/conf/:/usr/hdp/share/hst/hst-common/lib/* com.hortonworks.smartsense.anonymization.Main  -m /var/lib/smartsense/hst-agent/data/tmp/test.cluster-a-00045824-c-00000262_hdp_0_2017-01-15_12-12-12 -c /etc/hst/conf/hst-agent.ini failed ERROR anonymize.py:68 - Execution of command returned 1. Exception in thread "main" java.lang.IllegalArgumentException: Illegal group reference         at java.util.regex.Matcher.appendReplacement(Matcher.java:857)         at java.util.regex.Matcher.replaceAll(Matcher.java:955)         at java.lang.String.replaceAll(String.java:2223)         at com.hortonworks.smartsense.anonymization.BundleAnonymizer.handleApplyGroupPatternKeyValue(BundleAnonymizer.java:690)         at com.hortonworks.smartsense.anonymization.BundleAnonymizer.applyGroupPattern(BundleAnonymizer.java:673)         at com.hortonworks.smartsense.anonymization.BundleAnonymizer.applyPropertyRule(BundleAnonymizer.java:612)         at com.hortonworks.smartsense.anonymization.BundleAnonymizer.applyRules(BundleAnonymizer.java:393)         at com.hortonworks.smartsense.anonymization.BundleAnonymizer.anonymizeFolder(BundleAnonymizer.java:291)         at com.hortonworks.smartsense.anonymization.BundleAnonymizer.anonymizeFolder(BundleAnonymizer.java:259)         at com.hortonworks.smartsense.anonymization.BundleAnonymizer.anonymizeFolder(BundleAnonymizer.java:224)         at com.hortonworks.smartsense.anonymization.BundleAnonymizer.anonymize(BundleAnonymizer.java:160)         at com.hortonworks.smartsense.anonymization.Main.run(Main.java:82)         at com.hortonworks.smartsense.anonymization.Main.start(Main.java:210)         at com.hortonworks.smartsense.anonymization.Main.main(Main.java:294) ERROR AnonymizeBundleCommand.py:62 - Anonymization failed. Please check logs. Traceback (most recent call last):   File "/usr/hdp/share/hst/hst-agent/lib/hst_agent/command/AnonymizeBundleCommand.py", line 58, in execute     context['bundle_dir'] = anonymizer.anonymize(bundle_dir)   File "/usr/hdp/share/hst/hst-agent/lib/hst_agent/anonymize.py", line 69, in anonymize     raise Exception("Anonymization failed.") Exception: Anonymization failed. Diagnosis: The following xml property pattern: <property><name>PROP_NAME</name><value>VAL_TO_ANONYMIZE</value></property> is used to build the anonymized XML section. If "VAL_TO_ANONYMIZE" contains any reserved regex characters (such as '*') then the replace method will fail with an exception similar to the example above. Solution: For SmartSense version 1.3.1 there is a HOTFIX-651 patch available from support. In addition to the patch, users can also upgrade SmartSense to any 1.4.x version where the issue has also been fixed. If neither patching nor upgrading are options then as a temporary solution users can remove/replace the anonymization rule which has the offending regex reserved characters in its VAL_TO_ANONYMIZE definition. ... View more

Symptoms: Users may see an incorrect host domain name that may also include unexpected markers e.g. hadoopnode1.kjtijasdh.com , hadoopnode1.??kjtijasdh??.com Diagnosis: All data collected by HST Agents is anonymized via a process that uses a set of configurations along with anonymization rules to mask potentially sensitive data. The configurations include definitions for anonymization markers that are used to identify the start and end of a piece of data that has been anonymized via the rules. Certain anonymized content, such as host domain names, can later be deanonymized at the time of sending SmartSense recomendations so that the host domain is recognizable to the user. Typically the configured markers are unicode characters such as “¶” and “‡” which can can be incorrectly identified if the locale on the host running the HST Agent is not setup correctly which results in ?? as a default replacement. Solution: Making sure that you are using the same user that runs the HST agent on the problem host, display the locale setting with the following command: locale –a Look for the LANG setting; if LANG is not setup with a UTF8 variant then we must add an export statement setting LANG to the appropriate UTF8 encoding in the user's profile (same one that runs the HST Agent). The following is an example of this change for english/US: export LANG=en_US.utf8 The language might change from English to something else based on your locale but it has to be UTF8. Repeat the same change on all impacted hosts in your cluster. Once the change has been completed and sourced, please capture another SmartSense bundle to get new recommendations with the corrected host domain. ... View more