Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to change Cloudera services like kafka , spark configuration without web UI interface

How to change Cloudera services like kafka , spark configuration without web UI interface

Hi

 

I want to automate to change default configuration of kafka , yarn  using chef recipe . so I want to know that how can i change these services configuration in all hosts without using clouder Manger Web UI interface . and is any chef cookbook is available for cloudera configuration ?

 

 

Regards

Prateek

5 REPLIES 5
Highlighted

Re: How to change Cloudera services like kafka , spark configuration without web UI interface

Super Collaborator

You could use the CM REST API to change these config [1]

example adding HDFS dependency for Spark service using python CM API [2] [3]

 

Similar as above using curl [3]

 

$ curl -v -X PUT -u admin:pw -H 'Content-Type:application/json' -d '{
"items" : [ {
"name" : "hdfs_service",
"value" : "hdfs-1"
} ]
}' 'http://${your-scm-host}:7180/api/v10/cluster/[clustername]/service/[servicename]/config'

 

 

[1] http://cloudera.github.io/cm_api/apidocs/v11/path__clusters_-clusterName-_services_-serviceName-_con...

[2] https://github.com/cloudera/cm_api/blob/master/python/examples/auto-deploy/deploycloudera.py#L286-L2...

[3] https://github.com/cloudera/cm_api/blob/master/python/examples/auto-deploy/deploycloudera.py#L651

 

Highlighted

Re: How to change Cloudera services like kafka , spark configuration without web UI interface

 

Thanks for the information . it can solve my problem .

 

but i just want to know that 

Is it possible in cloudera  to directly modify conf files of kakfa , yarn etc . ? 

 

means can i chnage kafka properties in /etc/kafka/conf/server.properties file  and then restart my kafka in cloudera  with new configuration ?

 

Regards

Prateek

 

 

Highlighted

Re: How to change Cloudera services like kafka , spark configuration without web UI interface

Super Collaborator

Yes that is also possible, throught the use of the CM API. Details of how Cloudera Manager defines these configuration is detailed in [1] section "Configuration Management" and the API endpoint to restart a service is [2] also [3][4] to deploy the client configs.

 

 

 

[1] http://www.cloudera.com/content/www/en-us/documentation/enterprise/latest/topics/cm_intro_primer.htm...

[2] http://cloudera.github.io/cm_api/apidocs/v11/path__clusters_-clusterName-_services_-serviceName-_com...

[3] http://cloudera.github.io/cm_api/apidocs/v11/path__clusters_-clusterName-_commands_deployClientConfi...

[4] http://cloudera.github.io/cm_api/apidocs/v11/path__clusters_-clusterName-_services_-serviceName-_com...

Highlighted

Re: How to change Cloudera services like kafka , spark configuration without web UI interface

New Contributor

Hello Michaelis,
I also have the same question and could not understand from your answer what is the right approach.

 

I installed Kafka on Cloudera Manager using Parcels.
In my case, I want to change the kafka property "log.cleaner.enable" from "false" to "true".
This property is not supprted by Cloudera Manager configuration.

 

So I have two options:
1. update the file "/etc/kafka/conf/server.properties" and restart the kafka service from Cloudera Manager
2. add the value "log.cleaner.enable=true" to the Configuration key: "Kafka Broker Advanced Configuration Snippet (Safety Valve) for kafka.properties"

 

Which of the above is a better approach, and will they work?

Thank you,
Ran

Highlighted

Re: How to change Cloudera services like kafka , spark configuration without web UI interface

Super Collaborator

Since your cluster is managed by Cloudera Manager 2nd option is the right one, here's the example;

 

curl -X PUT -H 'Content-Type:application/json' -u admin:admin -d '{
  "items" : [ {
    "name" : "kafka.properties_role_safety_valve",
    "value" : "log.cleaner.enable=true\n"
  } ]
}' http://type-cm-hostname-here:7180/api/v11/clusters/{cluster-name}/services/{kafka-service-name}/roleConfigGroups/kafka-KAFKA_BROKER-BASE/config

 

Don't have an account?
Coming from Hortonworks? Activate your account here