Ambari allows its users to have different configurations for different hosts for different components via configuration groups. Ambari initially assigns all hosts in your cluster to one, default configuration group for each service you install. When using Configuration groups, it enforces configuration properties that allow override, based on installed components for the selected service and group. For more information on this please refer to: https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.2.0/bk_ambari-user-guide/content/using_host_config_groups.html Here we will try to make a very small change in "flume-conf" content and will apply it to specific host ("erie3.example.com") where as all the other hosts will be using the default "flume-conf" configuration. We can simply use the following ambari API to list all the config_groups. http://AMBARI_HOST:8080/api/v1/clusters/CLUSTER_NAME/config_groups Example: http://erie1.example.com:8080/api/v1/clusters/ErieCluster/config_groups . Step-1).
======== Lets see the current "flume-conf" from ambari which looks something like following and applied to all the Ambari Hosts: # Flume agent config
sandbox.sources = eventlog
sandbox.channels = file_channel
sandbox.sinks = sink_to_hdfs
# Define / Configure source
sandbox.sources.eventlog.type = exec
sandbox.sources.eventlog.command = tail -F /var/log/eventlog-demo.log
sandbox.sources.eventlog.restart = true
sandbox.sources.eventlog.batchSize = 1000
#sandbox.sources.eventlog.type = seq
# HDFS sinks
sandbox.sinks.sink_to_hdfs.type = hdfs
sandbox.sinks.sink_to_hdfs.hdfs.fileType = DataStream
sandbox.sinks.sink_to_hdfs.hdfs.path = /flume/events
sandbox.sinks.sink_to_hdfs.hdfs.filePrefix = eventlog
sandbox.sinks.sink_to_hdfs.hdfs.fileSuffix = .log
sandbox.sinks.sink_to_hdfs.hdfs.batchSize = 1000
# Use a channel which buffers events in memory
sandbox.channels.file_channel.type = file
sandbox.channels.file_channel.checkpointDir = /var/flume/checkpoint
sandbox.channels.file_channel.dataDirs = /var/flume/data
# Bind the source and sink to the channel
sandbox.sources.eventlog.channels = file_channel
sandbox.sinks.sink_to_hdfs.channel = file_channel . Step-2).
======== Now suppose for Host ("erie3.example.com") we want to run the flume with a slightly different properly like [sandbox.sources.eventlog.command = tail -F /var/log/eventlog-demo-new-location.log] So in order to achieve that we will need to create a "config_group" json data which we will need to push to Ambari. Here we will create a file like "/tmp/erie3_flume_conf.json" [
{
"ConfigGroup": {
"cluster_name": "ErieCluster",
"group_name": "cfg_group_test1",
"tag": "FLUME",
"description": "FLUME configs for Changes",
"hosts": [
{
"host_name": "erie3.example.com"
}
],
"desired_configs": [
{
"type": "flume-conf",
"tag": "nextgen1",
"properties": {
"content":
"
# Flume agent config\r\n
sandbox.sources = eventlog \r\n
sandbox.channels = file_channel \r\n
sandbox.sinks = sink_to_hdfs \r\n
\r\n
# Define / Configure source \r\n
sandbox.sources.eventlog.type = exec \r\n
sandbox.sources.eventlog.command = tail -F /var/log/eventlog-demo-new-location.log \r\n
sandbox.sources.eventlog.restart = true \r\n
sandbox.sources.eventlog.batchSize = 1000 \r\n
#sandbox.sources.eventlog.type = seq \r\n
\r\n
# HDFS sinks \r\n
sandbox.sinks.sink_to_hdfs.type = hdfs \r\n
sandbox.sinks.sink_to_hdfs.hdfs.fileType = DataStream \r\n
sandbox.sinks.sink_to_hdfs.hdfs.path = /flume/events \r\n
sandbox.sinks.sink_to_hdfs.hdfs.filePrefix = eventlog \r\n
sandbox.sinks.sink_to_hdfs.hdfs.fileSuffix = .log \r\n
sandbox.sinks.sink_to_hdfs.hdfs.batchSize = 2000 \r\n
# Use a channel which buffers events in memory \r\n
sandbox.channels.file_channel.type = file \r\n
sandbox.channels.file_channel.checkpointDir = /var/flume/checkpoint \r\n
sandbox.channels.file_channel.dataDirs = /var/flume/data \r\n
# Bind the source and sink to the channel \r\n
sandbox.sources.eventlog.channels = file_channel \r\n
sandbox.sinks.sink_to_hdfs.channel = file_channel \r\n
"
}
},
{
"type": "flume-env",
"tag": "nextgen1"
}
]
}
}
] **Notice:** If our configuration contains new line then the json data should use "\r\n" characters sequence.
- Also notice that the above JSON configuration is specified for host "erie3.example.com" as following, (However we can have more comma separated hosts there): . "hosts": [
{
"host_name": "erie3.example.com"
}
] - We have also specified the name for our config group ["group_name": "cfg_group_test1",] . Step-3).
======== Lets now PUT these changes to ambari and see if it works: curl -u admin:admin -H "X-Requested-By: ambari" -X POST -d @/Users/jsensharma/Cases/Articles/Flume_Config_Group/erie3_flume_conf.json http://erie1.example.com:8080/api/v1/clusters/ErieCluster/config_groups
OUTPUT:
$ curl -u admin:admin -H "X-Requested-By: ambari" -X POST -d @/Users/jsensharma/Cases/Articles/Flume_Config_Group/erie3_flume_conf.json http://erie1.example.com:8080/api/v1/clusters/ErieCluster/config_groups
{
"resources" : [
{
"href" : "http://erie1.example.com:8080/api/v1/clusters/ErieCluster/config_groups/252",
"ConfigGroup" : {
"id" : 252
}
}
]
} .
... View more
