Empty bundles captured in SmartSense post Ambari upgrade to 2.4.x

[#1]

SYMPTOM: When Ambari is upgraded to 2.4.x, SmartSense has to be upgraded before upgrading the HDP stack.

Failing to do so, would cause SmartSense to capture bundle with small size (in KB's) that has very less data.

ERROR:

ERROR 2016-12-01 23:02:02,279 anonymize.py:63 - Execution of command returned 1. Exception in thread "main" java.lang.NullPointerException
	at com.hortonworks.smartsense.anonymization.rules.RuleType.from(RuleType.java:47)
	at com.hortonworks.smartsense.anonymization.rules.Rules.createRule(Rules.java:161)
	at com.hortonworks.smartsense.anonymization.rules.Rules.createRules(Rules.java:148)
	at com.hortonworks.smartsense.anonymization.rules.Rules.createRules(Rules.java:132)
	at com.hortonworks.smartsense.anonymization.AnonymizerFactory.createAnonymizer(AnonymizerFactory.java:94)
	at com.hortonworks.smartsense.anonymization.Main.start(Main.java:202)
	at com.hortonworks.smartsense.anonymization.Main.main(Main.java:294)
ERROR 2016-12-01 23:02:02,280 AnonymizeBundleCommand.py:60 - Anonymization failed. Please check logs.
Traceback (most recent call last):
  File "/usr/hdp/share/hst/hst-agent/lib/hst_agent/command/AnonymizeBundleCommand.py", line 56, in execute
    context['bundle_dir'] = anonymizer.anonymize(bundle_dir)
  File "/usr/hdp/share/hst/hst-agent/lib/hst_agent/anonymize.py", line 64, in anonymize
    raise Exception("Anonymization failed.")

ROOT CAUSE:

When HDP is upgraded to 2.5.x, before following the SmartSense upgrade procedure, the smartsense-hst package also gets upgraded to 1.3.x. With the old configurations and Anonymization rules, any bundles captured would be either empty or contains very less data.

RESOLUTION: Follow the below steps to complete the SmartSense upgrade to 1.3.0.

• Step 1

Using Ambari UI, replace the SmartSense anonymization rules content (Under Services - SmartSense - Config - Data Capture ) to following data.

• {
  "rules":[
    {
      "name":"ip_address",
      "ruleId": "Pattern",
      "path":null,
      "pattern": "([^a-z0-9\\.]|^)[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}([^a-z0-9\\.\\-]|(\\.[^0-9])|$)",
      "extract": "[ :\\/]?([0-9\\.]+)[ :\\/]?",
      "excludes": ["hdp-select*.*", "*version.txt"],
      "shared":true
    },
    {
      "name":"domain",
      "ruleId": "Domain",
      "path":null,
      "pattern": "$DOMAIN_RULE$",
      "shared":true
    },
    {
      "name":"delete_oozie_jdbc_password",
      "ruleId": "Property",
      "path":"oozie-site.xml",
      "property": "oozie.service.JPAService.jdbc.password",
      "operation":"REPLACE",
      "value":"Hidden"
    },
    {
      "name":"delete_sqoop_metastore_password",
      "ruleId": "Property",
      "path":"sqoop-site.xml",
      "property": "sqoop.metastore.client.autoconnect.password",
      "operation":"REPLACE",
      "value":"Hidden"
    },
    {
      "name":"delete_hive_metastore_password",
      "ruleId": "Property",
      "path":"hive-site.xml",
      "property": "javax.jdo.option.ConnectionPassword",
      "operation":"REPLACE",
      "value":"Hidden"
    },
    {
      "name":"delete_s3_accesskey",
      "ruleId": "Property",
      "path":"core-site.xml",
      "property": "fs.s3.awsAccessKeyId",
      "operation":"REPLACE",
      "value":"Hidden"
    },
    {
      "name":"delete_s3_secret_accesskey",
      "ruleId": "Property",
      "path":"core-site.xml",
      "property": "fs.s3.awsSecretAccessKey",
      "operation":"REPLACE",
      "value":"Hidden"
    },
    {
      "name":"delete_s3n_accesskey",
      "ruleId": "Property",
      "path":"core-site.xml",
      "property": "fs.s3n.awsAccessKeyId",
      "operation":"REPLACE",
      "value":"Hidden"
    },
    {
      "name":"delete_s3n_secret_accesskey",
      "ruleId": "Property",
      "path":"core-site.xml",
      "property": "fs.s3n.awsSecretAccessKey",
      "operation":"REPLACE",
      "value":"Hidden"
    },
    {
      "name":"delete_azure_account_key",
      "ruleId": "Property",
      "path":"core-site.xml",
      "property": "fs.azure.account.key.*",
      "operation":"REPLACE",
      "value":"Hidden"
    },
    {
      "name":"delete_ldap_password",
      "ruleId": "Property",
      "path":"core-site.xml",
      "property": "hadoop.security.group.mapping.ldap.bind.password",
      "operation":"REPLACE",
      "value":"Hidden"
    },
    {
      "name":"hide_ssl_client_keystore_pwd",
      "ruleId": "Property",
      "path":"ssl-client.xml",
      "property": "ssl.client.keystore.password",
      "operation":"REPLACE",
      "value":"Hidden"
    },
    {
      "name":"hide_ssl_client_truststore_pwd",
      "ruleId": "Property",
      "path":"ssl-client.xml",
      "property": "ssl.client.truststore.password",
      "operation":"REPLACE",
      "value":"Hidden"
    },
    {
      "name":"hide_ssl_server_keystore_keypwd",
      "ruleId": "Property",
      "path":"ssl-server.xml",
      "property": "ssl.server.keystore.keypassword",
      "operation":"REPLACE",
      "value":"Hidden"
    },
    {
      "name":"hide_ssl_server_keystore_pwd",
      "ruleId": "Property",
      "path":"ssl-server.xml",
      "property": "ssl.server.keystore.password",
      "operation":"REPLACE",
      "value":"Hidden"
    },
    {
      "name":"hide_ssl_server_truststore_pwd",
      "ruleId": "Property",
      "path":"ssl-server.xml",
      "property": "ssl.server.truststore.password",
      "operation":"REPLACE",
      "value":"Hidden"
    },
    {
      "name":"hide_oozie_pwd_in_java_process_info",
      "ruleId": "Pattern",
      "path":"java_process.txt",
      "pattern": "oozie.https.keystore.pass=([^ ]*)",
      "extract": "=([^ ]*)",
      "shared":false
    },
    {
      "name":"hide_oozie_pwd_in_process_info",
      "ruleId": "Pattern",
      "path":"pid.txt",
      "pattern": "oozie.https.keystore.pass=([^ ]*)",
      "extract": "=([^ ]*)",
      "shared":false
    },
    {
      "name":"hide_oozie_pwd_in_ambariagent_log",
      "ruleId": "Pattern",
      "path":"ambari-agent.log",
      "pattern": "oozie.https.keystore.pass=([^ ]*)",
      "extract": "=([^ ]*)",
      "shared":false
    },
    {
      "name":"delete_oozie_https_keystore_pass",
      "ruleId": "Pattern",
      "path":"oozie-env.cmd",
      "pattern":"OOZIE_HTTPS_KEYSTORE_PASS=([^ ]*)",
      "extract": "=([^ ]*)",
      "shared":false
    },
    {
      "name":"java_process_ganglia_password",
      "ruleId": "Pattern",
      "path":"java_process.txt",
      "pattern":"ganglia_password=([^ ]*)",
      "extract": "=([^ ]*)",
      "shared":false
    },
    {
      "name":"hide_ssn_from_logs",
      "ruleId": "Pattern",
      "path":"*\\.log*",
      "pattern": "(^|[^0-9x])[0-9x]{3}-[0-9x]{2}-[0-9]{4}($|[^0-9x])",
      "extract": "(?<![0-9x])([0-9x-]+)(?![0-9x])",
      "shared":true
    },
    {
      "name":"hide_credit_card_from_logs",
      "ruleId": "Pattern",
      "path":"*\\.log*",
      "pattern": "(^|[^0-9x])(18|21|3[04678]|4[0-9x]|5[1-5]|60|65)[0-9x]{2}[- ]([0-9x]{4}[- ]){2}[0-9]{0,4}($|[^0-9x])",
      "extract": "(?<![0-9x])([0-9x -]+)(?![0-9x])",
      "shared":true
    },
    {
      "name": "hide_knox_ldap_password",
      "ruleId": "Property",
      "path": "services/KNOX/components/KnoxGateway/DEFAULT/conf/topologies/*.xml",
      "property": "main.ldapRealm.contextFactory.systemPassword",
      "parentTag": "param",
      "operation": "REPLACE",
      "value": "Hidden"
    },
    {
      "name": "delete_kms_https_keystore_pass",
      "ruleId": "Property",
      "path": "kms-log4j.properties",
      "property": "KMS_SSL_KEYSTORE_PASS",
      "operation": "REPLACE",
      "value": "Hidden"
    },
    {
      "name": "hide_kms_keystore_provider_pwd",
      "ruleId": "Property",
      "path": "kms-site.xml",
      "property": "hadoop.security.keystore.JavaKeyStoreProvider.password",
      "operation": "REPLACE",
      "value": "Hidden"
    },
    {
      "name": "hide_ranger_https_keystore_pwd",
      "ruleId": "Property",
      "path": "xasecure-policymgr-ssl.xml",
      "property": "xasecure.policymgr.clientssl.keystore.password",
      "operation": "REPLACE",
      "value": "Hidden"
    },
    {
      "name": "hide_ranger_https_truststore_pwd",
      "ruleId": "Property",
      "path": "xasecure-policymgr-ssl.xml",
      "property": "xasecure.policymgr.clientssl.truststore.password",
      "operation": "REPLACE",
      "value": "Hidden"
    },
    {
      "name":"delete_ranger_webserver_https_keystore_pwd",
      "ruleId": "Property",
      "path":"ranger_webserver.properties",
      "property":"HTTPS_KEYSTORE_PASS",
      "operation": "REPLACE",
      "value": "Hidden"
    },
    {
      "name":"delete_ranger_webserver_attrib_https_keystore_pwd",
      "ruleId": "Property",
      "path":"ranger_webserver.properties",
      "property":"https.attrib.keystorePass",
      "operation": "REPLACE",
      "value": "Hidden"
    },
    {
      "name":"delete_ranger_keystore_pwd",
      "ruleId": "Property",
      "path":"unixauthservice.properties",
      "property":"keyStorePassword",
      "operation": "REPLACE",
      "value": "Hidden"
    },
    {
      "name":"delete_ranger_truststore_pwd",
      "ruleId": "Property",
      "path":"unixauthservice.properties",
      "property":"trustStorePassword",
      "operation": "REPLACE",
      "value": "Hidden"
    },
    {
      "name": "hide_ambari_ssl_truststore_pwd",
      "ruleId": "Property",
      "path": "ambari.properties",
      "property": "ssl.trustStore.password",
      "operation": "REPLACE",
      "value": "Hidden"
    }
  ]
  }

• Step 2 (This assumes that upgrade steps are not run before.)

Update Capture levels configurations in Ambari by running following command

• # /var/lib/ambari-server/resources/scripts/configs.sh -u <ambari_user_id> -p <ambari_userid_password> set localhost <hdp_cluster_name> capture-levels capture-levels-content '[\n    {\n      \"name\": \"L1\",\n      \"description\": \"Configurations\",\n      \"filter\": [\"CONFIGS\"]\n    },\n    {\n      \"name\": \"L2\",\n      \"description\": \"Configurations, reports and metrics\",\n      \"filter\": [\"CONFIGS\", \"REPORTS\", \"METRICS\"]\n    },\n    {\n      \"name\": \"L3\",\n      \"description\": \"Configurations, reports, metrics and Logs\",\n      \"filter\": [\"CONFIGS\", \"REPORTS\", \"METRICS\", \"LOGS\", \"DIAGNOSTICS\"]\n    }\n]'
  

If the above command fails, then manually update /etc/hst/conf/capture_levels.json file on HST server with below content

• [
      {
        "name": "L1",
        "description": "Configurations",
        "filter": ["CONFIGS"]
      },
      {
        "name": "L2",
        "description": "Configurations, reports and metrics",
        "filter": ["CONFIGS", "REPORTS", "METRICS"]
      },
      {
        "name": "L3",
        "description": "Configurations, reports, metrics and Logs",
        "filter": ["CONFIGS", "REPORTS", "METRICS", "LOGS", "DIAGNOSTICS"]
      }
  ]

• Step 3
  • Restart SmartSense Services via Ambari, to propate the above changes to all the agents.
• Step 4
  • Confirm smartsense-hst rpm package is upgraded to version 1.3.0

• # rpm -qi smartsense-hst

  • On Ambari server host, as root user, run the following command

• # hst upgrade-ambari-service

• Step 5
  • Restart Ambari Server

• # ambari-server restart

• Step 6
  • Restart SmartSense service via Ambari to propagate final changes to agent hosts.

• Step 7
  • Start Bundle Capture from Ambari SmartSense view and confirm the contents and size of the bundle.