(ambari) Append to -env templates (e.g. hadoop-env) cleanly?

[#1]

How can we append to '-env' in a clean programatic way?

2 scenarios:

1. Blueprints: How to append to 'hadoop-env' at cluster build?
   1. It appears you would have to set the entire block. Which means fetching from an existing cluster and setting the entire thing. This is not reliable, nor is it sustainable since the defaults may change between versions of Ambari/HDP.
2. How to change programmatically after cluster build?
   1. Ideally you would be able to append to the existing configuration.

[#2 · Accepted solution]

Regarding Option1: I was under the impression that you can use xxxx-env blocks without defining the whole block within blueprints, e.g. I was using a blueprint recently to add the existing MySQL configuration to "hive-env". But I didn't check hive-env after the blueprint installation via the Ambari API, so I am not sure if the hive-env was messed up or not (I'll run another test in a week or so), Hive did start though.

Regarding Option 2: You can use a simple Python script, here are the necessary steps

1. Find current config version ("tag"-field), e.g. cluster-env

curl -H "X-Requested-By:ambari" -u admin:your-pw -X GET "http://c6601.ambari.apache.org:8080/api/v1/clusters/bigdata?fields=Clusters/desired_configs/cluster-env"

Response:

{
  "href" : "http://c6601.ambari.apache.org:8080/api/v1/clusters/bigdata?fields=Clusters/desired_configs/cluster-env",
  "Clusters" : {
    "cluster_name" : "bigdata",
    "version" : "HDP-2.3",
    "desired_configs" : {
      "cluster-env" : {
        "tag" : "version1434441386979",
        "user" : "admin",
        "version" : 7
      }
    }
  }
}

The tag-field is import => version1434441386979

2. Export current config (example result below)

curl -H "X-Requested-By:ambari" -u admin:your-pw -X GET "http://c6601.ambari.apache.org:8080/api/v1/clusters/bigdata/configurations?type=cluster-env&tag=version1434441386979" 

Response (truncated):

{
   "href":"http://c6601.ambari.apache.org:8080/api/v1/clusters/bigdata/configurations?type=cluster-env&tag=version1434441386979",
   "items":[
      {
         "href":"http://c6601.ambari.apache.org:8080/api/v1/clusters/bigdata/configurations?type=cluster-env&tag=version1434441386979",
         "tag":"version1434441386979",
         "type":"cluster-env",
         "version":7,
         "Config":{
            "cluster_name":"bigdata"
         },
         "properties":{
            ...
            "smokeuser_keytab":"/etc/security/keytabs/smokeuser.headless.keytab",
            "smokeuser_principal_name":"ambari-qa@EXAMPLE.COM",
            "sqoop_tar_destination_folder":"hdfs:///hdp/apps/{{ hdp_stack_version }}/sqoop/",
            "sqoop_tar_source":"/usr/hdp/current/sqoop-client/sqoop.tar.gz",
            "tez_tar_destination_folder":"hdfs:///hdp/apps/{{ hdp_stack_version }}/tez/",
            "tez_tar_source":"/usr/hdp/current/tez-client/lib/tez.tar.gz",
            "user_group":"hadoop"
         }
      }
   ]
}

3. Prepare the new configuration by copying all properties from above in the following template.

Template:

{ 
  "Clusters" : {
    "desired_configs" : { 
      "type" : "cluster-env", 
      "tag" : "version<INSERT_CURRENT_TIMESTAMP_IN_MILLISECONDS>", 
      "properties" : { 
        <INSERT_EXPORTED_CONFIGURATION_PROPERTIES_FROM_ABOVE>
      }
    }
  }
}

For example, lets say my user_group is not hadoop anymore, from now on its horton

{ 
  "Clusters" : {
    "desired_configs" : { 
      "type" : "cluster-env", 
      "tag" : "version<INSERT_CURRENT_TIMESTAMP_IN_MILLISECONDS>", 
      "properties" : { 
        ...
        "smokeuser_keytab" : "/etc/security/keytabs/smokeuser.headless.keytab",
        "smokeuser_principal_name" : "ambari-qa@EXAMPLE.COM",
        "sqoop_tar_destination_folder" : "hdfs:///hdp/apps/{{ hdp_stack_version }}/sqoop/",
        "sqoop_tar_source" : "/usr/hdp/current/sqoop-client/sqoop.tar.gz",
        "tez_tar_destination_folder" : "hdfs:///hdp/apps/{{ hdp_stack_version }}/tez/",
        "tez_tar_source" : "/usr/hdp/current/tez-client/lib/tez.tar.gz",
        "user_group" : "horton"
      }
    }
  }
}

5. Final step: Upload new configuration to cluster

curl -H "X-Requested-By:ambari" -u admin:your-pw -X PUT "http://c6601.ambari.apache.org:8080/api/v1/clusters/bigdata" -d @<JSON_PAYLOAD_FROM_ABOVE>

DONE 🙂

Hope it helps

Jonas

[#3]

@Jonas Straub

1. Let me know what you find. I bet you wiped it out. But would love if it worked properly.

[#4]

@Sean Roberts Just finalized my blueprint installation with a modified hive-env. I basically just configured an existing MySQL database in hive-env in the blueprint. Ambari added the rest of the hive-env variables to the configuration, so there are no config values missing.

[#5]

@Jonas Straub

2. Yep, that's what I've been doing. It works, but configuration changes shouldn't require such complexity. The API needs to be more human understandable.

An alternative is to be really hacky with 'configs.sh' and 'sed':

cd /tmp
configssh="/var/lib/ambari-server/resources/scripts/configs.sh"

${configssh} get localhost mycluster hive-env \
    | sed -e '1,3d' \
    -e '/^"content" : / s#",$#\\n\\n WHATEVER-YOU-WANT-TO-ADD \\n",#' \
    > /tmp/hive-env.json
${configssh} set localhost mycluster hive-env /tmp/hive-env.json

[#6]

@Jonas Straub :

>> Regarding Option1: I was under the impression that you can use xxxx-env blocks without defining the whole block within blueprints, e.g. I was using a blueprint recently to add the existing MySQL configuration to "hive-env". But I didn't check hive-env after the blueprint installation via the Ambari API, so I am not sure if the hive-env was messed up or not (I'll run another test in a week or so), Hive did start though.

Did you ever get to test this? Seems like it is not working.

I tried using defining "content" property for hadoop-env.sh for just HADOOP_NAMENODE_OPTS and HADOOP_DATANODE_OPTS as follows. hadoop-env.sh resulted in just only those two lines and nothing else.

"hadoop-env": {

"properties": {

"content" : "export HADOOP_NAMENODE_OPTS=\"-server -XX:ParallelGCThreads=8 -XX:+UseConcMarkSweepGC-XX:ErrorFile={{hdfs_log_dir_prefix}}/$USER/hs_err_pid%p.log -XX:NewSize={{namenode_opt_newsize}} -XX:MaxNewSize={{namenode_opt_maxnewsize}} -XX:PermSize={{namenode_opt_permsize}} -XX:MaxPermSize={{namenode_opt_maxpermsize}} -Xloggc:{{hdfs_log_dir_prefix}}/$USER/gc.log-`date +'%Y%m%d%H%M'` -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xms{{namenode_heapsize}} -Xmx{{namenode_heapsize}} -Dhadoop.security.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,DRFAAUDIT ${HADOOP_NAMENODE_OPTS}\"\n export HADOOP_DATANODE_OPTS=\"-server -XX:ParallelGCThreads=4 -XX:+UseConcMarkSweepGC -XX:ErrorFile={{hdfs_log_dir_prefix}}/$USER/hs_err_pid%p.log -XX:NewSize=200m-XX:MaxNewSize=200m -XX:PermSize=128m -XX:MaxPermSize=256m -Xloggc:{{hdfs_log_dir_prefix}}/$USER/gc.log-`date +'%Y%m%d%H%M'` -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xms{{dtnode_heapsize}}-Xmx{{dtnode_heapsize}} -Dhadoop.security.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,DRFAAUDIT ${HADOOP_DATANODE_OPTS}\"\n",

"namenode_opt_maxnewsize": "361m",

"namenode_opt_newsize": "361m",

"namenode_heapsize": "2887m",

"dtnode_heapsize": "2887m",

"dfs_ha_enabled": "true",

"hdfs_log_dir_prefix": "/data/var/log/hadoop",

"mapred_log_dir_prefix": "/data/var/log/hadoop-mapreduce",

"yarn_log_dir_prefix": "/data/var/log/hadoop-yarn",

"hive_log_dir": "/data/var/log/hive",

"zk_log_dir": "/data/var/log/zookeeper",

"metrics_monitor_log_dir": "/data/var/log/ambari-metrics-collector",

"metrics_collector_log_dir": "/data/var/log/ambari-metrics-monitor",

"kafka_log_dir": "/data/var/log/kafka",

"spark_log_dir": "/data/var/log/spark"

}

}