SmartSense agent fails with anonymization error during capture

[#1]

Symptoms:

HST Bundle capture fails during the anonymization process with an "Illegal group reference" message exception found in the hst-agent logs:

ERROR shell.py:95 - Execution of command returned 1. Exception in thread "main" java.lang.IllegalArgumentException: Illegal group reference 
        at java.util.regex.Matcher.appendReplacement(Matcher.java:857) 
        at java.util.regex.Matcher.replaceAll(Matcher.java:955) 
        at java.lang.String.replaceAll(String.java:2223) 
        at com.hortonworks.smartsense.anonymization.BundleAnonymizer.handleApplyGroupPatternKeyValue(BundleAnonymizer.java:690) 
        at com.hortonworks.smartsense.anonymization.BundleAnonymizer.applyGroupPattern(BundleAnonymizer.java:673) 
        at com.hortonworks.smartsense.anonymization.BundleAnonymizer.applyPropertyRule(BundleAnonymizer.java:612) 
        at com.hortonworks.smartsense.anonymization.BundleAnonymizer.applyGroupPattern(BundleAnonymizer.java:673) 
        at com.hortonworks.smartsense.anonymization.BundleAnonymizer.applyPropertyRule(BundleAnonymizer.java:612) 
        at com.hortonworks.smartsense.anonymization.BundleAnonymizer.applyRules(BundleAnonymizer.java:393) 
        at com.hortonworks.smartsense.anonymization.BundleAnonymizer.anonymizeFolder(BundleAnonymizer.java:291) 
        at com.hortonworks.smartsense.anonymization.BundleAnonymizer.anonymizeFolder(BundleAnonymizer.java:259) 
        at com.hortonworks.smartsense.anonymization.BundleAnonymizer.anonymizeFolder(BundleAnonymizer.java:224) 
        at com.hortonworks.smartsense.anonymization.Main.run(Main.java:82) 
        at com.hortonworks.smartsense.anonymization.Main.start(Main.java:210) 
        at com.hortonworks.smartsense.anonymization.Main.main(Main.java:294) 
ERROR anonymize.py:67 - Execution of script /usr/lib/jvm/java/bin/java -Xmx2048m -Xms1024m -Dlog.file.name=anonymization.log -Djava.io.tmpdir=/var/lib/smartsense/hst-agent/data/tmp -cp :/etc/hst/conf/:/usr/hdp/share/hst/hst-common/lib/* com.hortonworks.smartsense.anonymization.Main  -m /var/lib/smartsense/hst-agent/data/tmp/test.cluster-a-00045824-c-00000262_hdp_0_2017-01-15_12-12-12 -c /etc/hst/conf/hst-agent.ini failed 
ERROR anonymize.py:68 - Execution of command returned 1. Exception in thread "main" java.lang.IllegalArgumentException: Illegal group reference 
        at java.util.regex.Matcher.appendReplacement(Matcher.java:857) 
        at java.util.regex.Matcher.replaceAll(Matcher.java:955) 
        at java.lang.String.replaceAll(String.java:2223) 
        at com.hortonworks.smartsense.anonymization.BundleAnonymizer.handleApplyGroupPatternKeyValue(BundleAnonymizer.java:690) 
        at com.hortonworks.smartsense.anonymization.BundleAnonymizer.applyGroupPattern(BundleAnonymizer.java:673) 
        at com.hortonworks.smartsense.anonymization.BundleAnonymizer.applyPropertyRule(BundleAnonymizer.java:612) 
        at com.hortonworks.smartsense.anonymization.BundleAnonymizer.applyRules(BundleAnonymizer.java:393) 
        at com.hortonworks.smartsense.anonymization.BundleAnonymizer.anonymizeFolder(BundleAnonymizer.java:291) 
        at com.hortonworks.smartsense.anonymization.BundleAnonymizer.anonymizeFolder(BundleAnonymizer.java:259) 
        at com.hortonworks.smartsense.anonymization.BundleAnonymizer.anonymizeFolder(BundleAnonymizer.java:224) 
        at com.hortonworks.smartsense.anonymization.BundleAnonymizer.anonymize(BundleAnonymizer.java:160) 
        at com.hortonworks.smartsense.anonymization.Main.run(Main.java:82) 
        at com.hortonworks.smartsense.anonymization.Main.start(Main.java:210) 
        at com.hortonworks.smartsense.anonymization.Main.main(Main.java:294) 
ERROR AnonymizeBundleCommand.py:62 - Anonymization failed. Please check logs. 
Traceback (most recent call last): 
  File "/usr/hdp/share/hst/hst-agent/lib/hst_agent/command/AnonymizeBundleCommand.py", line 58, in execute 
    context['bundle_dir'] = anonymizer.anonymize(bundle_dir) 
  File "/usr/hdp/share/hst/hst-agent/lib/hst_agent/anonymize.py", line 69, in anonymize 
    raise Exception("Anonymization failed.") 
Exception: Anonymization failed. 

Diagnosis:

The following xml property pattern:

<property><name>PROP_NAME</name><value>VAL_TO_ANONYMIZE</value></property>

is used to build the anonymized XML section. If "VAL_TO_ANONYMIZE" contains any reserved regex characters (such as '*') then the replace method will fail with an exception similar to the example above.

Solution:

For SmartSense version 1.3.1 there is a HOTFIX-651 patch available from support. In addition to the patch, users can also upgrade SmartSense to any 1.4.x version where the issue has also been fixed. If neither patching nor upgrading are options then as a temporary solution users can remove/replace the anonymization rule which has the offending regex reserved characters in its VAL_TO_ANONYMIZE definition.