Created 07-19-2016 02:22 PM
Hi,
I am trying to deploy a HDP cluster via Ambari on a number of VM's that are each running a different version of zookeeper already.
As expected, I am encountering errors that the zookeeper client port (:2181) is already occupied when I try to deploy zookeeper via Ambari.
I tried changing the client port to an unoccupied port in /zoo.cfg in the *Customize Services* of Ambari deployment. Doing this I was able to successfully deploy Zookeeper, but the services that connect to zookeeper are still looking for it at :2181. An example of this is below:
"kafka-broker" : { "properties" : { ... "zookeeper.connect" : "%HOSTGROUP::host_group_1%:2181", ... } }
I would prefer not to manually change zookeepers client port on every single service that uses zookeeper. Is there an easier alternative?
Created 07-20-2016 08:09 AM
Please post it as an idea in HCC / file an enhancement request with Apache / Hortonworks.
Created on 07-19-2016 08:21 PM - edited 08-19-2019 01:50 AM
You can go to each service configuration in Ambari UI and change the port number.
For example for Kafka, see below screenshot
Created 07-19-2016 08:57 PM
Thanks for getting back to me!
I know that I can manually change the port number for each service, I was wondering if there was an alternative to doing that (I don't know much about Zookeeper).
The reason I ask is because I am working on an automated deployment of HDP, and manually changing the port (even if I did so within a Blueprint) gets in the way of that.
Created 07-19-2016 09:02 PM
@John Martin - If you are going to use blueprint then I believe mentioning port in the configuration is the only option.
Created 07-20-2016 02:50 AM
@Kuldeep Kulkarni each service configuration can be updated by using API. Take a look here. You can also make changes in the blueprint.
Configuration update involves the following steps:
You can use APIs or a wrapper script (/var/lib/ambari-server/resources/scripts/configs.sh) to edit configurations.
Starting 1.4.2/1.4.3 you will have to add -H option to the curl calls. E.g. -H "X-Requested-By: ambari"
1. Find the latest version of the config type that you need to update.
curl -u admin:admin -X GET http: //AMBARI_SERVER_HOST:8080/api/v1/clusters/CLUSTER_NAME?fields=Clusters/desired_configs
Sample OUTPUT
{
"href" : "http://AMBARI_SERVER_HOST:8080/api/v1/clusters/CLUSTER_NAME?fields=Clusters/desired_configs" ,
"Clusters" : {
"cluster_name" : "CLUSTER_NAME" ,
"version" : "HDP-2.0.6" ,
"desired_configs" : {
...
"mapred-site" : {
"user" : "admin" ,
"tag" : "version1384716039631"
}
...
}
}
}
|
2. Read the config type with correct tag
curl -u admin:admin "http://AMBARI_SERVER_HOST:8080/api/v1/clusters/CLUSTER_NAME/configurations?type=mapred-site&tag=vers...
Sample OUTPUT
{
"href" : "http://AMBARI_SERVER_HOST:8080/api/v1/clusters/CLUSTER_NAME/configurations?type=mapred-site&tag=vers... ,
"items" : [
{
"href" : "http://AMBARI_SERVER_HOST:8080/api/v1/clusters/CLUSTER_NAME/configurations?type=mapred-site&tag=vers... ,
"tag" : "version1384716039631" ,
"type" : "mapred-site" ,
"Config" : {
"cluster_name" : "CLUSTER_NAME"
},
"properties" : {
... THESE ARE THE PROPERTY KEY-VALUE PAIRS ...
}
}]
}
|
3a. Save a new version of the config and apply it (see 3b for doing it using one call)
curl --user admin:admin -i -X POST -d '{"type": "mapred-site", "tag": "version1384716041120", "properties" : {"mapreduce.admin.map.child.java.opts" : "-Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN",...}}' http: //AMBARI_SERVER_HOST:8080/api/v1/clusters/CLUSTER_NAME/configurations
curl --user admin:admin -i -X PUT -d '{"Clusters":{"desired_config" : {"type": "mapred-site", "tag": "version1384716041120"}}}' http: //AMBARI_SERVER_HOST:8080/api/v1/clusters/CLUSTER_NAME
|
3b. Save a new version of the config and apply it using one call
curl --user admin:admin -i -X PUT -d '{"Clusters":{"desired_config" : {"type": "mapred-site", "tag": "version1384716041120", "properties" : {...}}}}' http: //AMBARI_SERVER_HOST:8080/api/v1/clusters/CLUSTER_NAME
|
4. Restart all components or services to have the config change take effect E.g. Stop and Start a service
curl --user admin:admin -i -X PUT -d '{"RequestInfo": {"context": "Stop HDFS"}, "ServiceInfo": {"state": "INSTALLED"}}' http: //AMBARI_SERVER_HOST:8080/api/v1/clusters/CLUSTER_NAME/services/HDFS
curl --user admin:admin -i -X PUT -d '{"RequestInfo": {"context": "Start HDFS"}, "ServiceInfo": {"state": "STARTED"}}' http: //AMBARI_SERVER_HOST:8080/api/v1/clusters/CLUSTER_NAME/services/HDFS |
Created 07-20-2016 09:49 PM
@Sunile Manjee - Thanks for the info, I'm aware that we can do this using API, as @John Martin wanted to do this while installing HDP cluster hence suggested to put in configs section of ambari blueprint.
Created 07-20-2016 08:09 AM
Please post it as an idea in HCC / file an enhancement request with Apache / Hortonworks.