Failed to get clientConfig via API /clusters/{clusterName}/services/{serviceName}/clientConfig

[#1]

I upgraded Cloud Manager from 5.10.0 to 6.3.0, then i tried to run 'wget http://10.0.0.209:7180/api/v13/clusters/{clusterName}/services/{serviceName}/clientConfig', got a error 404. Checked the log file of cloudera-scm-server, got the following:

2020-09-15 04:11:57,977 INFO scm-web-16747:com.cloudera.api.ApiExceptionMapper: Exception caught in API invocation. Msg:Service 'yarn' not found in cluster 'platform-pod10-eason-cdh'.
java.util.NoSuchElementException: Service 'yarn' not found in cluster 'platform-pod10-eason-cdh'.
at com.cloudera.api.dao.impl.ManagerDaoBase.findService(ManagerDaoBase.java:484)
......

but I checked the services via accessing API /clusters/{clusterName}/services/{serviceName}, actually the service 'yarn' is in this cluster, ref:

[root@rhel76x64 ~]# curl http://<user>:<password>@10.0.0.209:7180/api/v13/clusters/platform-pod10-eason-cdh/services/yarn/
{
"healthChecks" : [ {
"name" : "YARN_JOBHISTORY_HEALTH",
"summary" : "GOOD",
"suppressed" : false
}, {
"name" : "YARN_NODE_MANAGERS_HEALTHY",
"summary" : "GOOD",
"suppressed" : false
}, {
"name" : "YARN_RESOURCEMANAGERS_HEALTH",
"summary" : "GOOD",
"suppressed" : false
}, {
"name" : "YARN_USAGE_AGGREGATION_HEALTH",
"summary" : "DISABLED",
"suppressed" : false
} ],
"maintenanceOwners" : [ ],
"name" : "yarn",
"type" : "YARN",
"clusterRef" : {
"clusterName" : "cluster"
},
"serviceUrl" : "http://ip-10-0-0-209.us-west-2.compute.internal:7180/cmf/serviceRedirect/yarn",
"roleInstancesUrl" : "http://ip-10-0-0-209.us-west-2.compute.internal:7180/cmf/serviceRedirect/yarn/instances",
"serviceState" : "STARTED",
"healthSummary" : "GOOD",
"configStalenessStatus" : "FRESH",
"clientConfigStalenessStatus" : "FRESH",
"maintenanceMode" : false,
"displayName" : "YARN (MR2 Included)",
"entityStatus" : "GOOD_HEALTH"

In addition, i can get clientConfig successfully by running 'wget http://10.0.0.209:7180/api/v13/clusters/{clusterName}/services/{serviceName}/clientConfig --user=xxx --password=xxx --auth-no-challenge', but base on the specification of API, this API does not require any authentication.

Does anyone have a clue about it? Thanks!

[#2 · Accepted solution]

Hello @iEason8 ,

you've got a good point. The main page for the doc (which I've cited earlier) says you need authentication for API calls for CM6 however, the section you've found says explicitly it does not require authentication.

So I've done some research in our records and found that the desired behaviour for CM6 is to enforce authentication for API calls (internal jira reference OPSAPS-44459) and that there is a bug in our doc (internal jira reference DOCS-4659) that explains "Doc wrongly suggests that clientConfig API call does not require any authentication" and this jira is unresolved at this moment.

Sorry for the inconvenience caused. You need to get authenticated to be able to use the API in CM6.

Kind regards:

Ferenc

[#3]

Hello @iEason8 ,

thank you for reaching out to Community! I've noticed that you are using v13 API for CM6.3, however CM6.3 uses v33 API. Please see this reference doc for the API.

If you adjust your request accordingly to the new API, do you still have the issue?

Kind regards:
Ferenc

[#4]

@Bender yes, the issue still exists even if i use v33 API.

The strange points are:

1. This API /clusters/{clusterName}/services/{serviceName}/clientConfig is supposed to not require any authentication, but i need to specify credential to access it succesfully via curl/wget.
2. If i don't specify credential, the error message is "Service 'yarn' not found in cluster 'platform-pod10-eason-cdh'.", but 'yarn' is actually in the cluster, which is confusing.

Is there a switch of the authentication (e.g. turn on/off)? or did i miss anything? Thanks in advance!

[#5]

@Bender Do you (or anyone) have a Cloudera Manager 6.3.x installed? If yes, could you please help to try to run 'wget http://<cm_server_host>:7180/api/v33/clusters/{clusterName}/services/{serviceName}/clientConfig' to check if it can work as expected? Thanks a lot!

[#6]

Hi @iEason8 ,

I've managed to get the config and the clientConfig too using curl.

The cluster name in my case was "Cluster 1", the service name was "KAFKA-1", so had to URL encode the space. I've highlighted the parts in bold you will need to change. Make sure the user you use have the necessary privileges to get the configs:

"The Cloudera Manager API uses HTTP basic access authentication. It accepts the same user credentials as the web interface. Different users may have different levels of access, as defined by their roles. (See the user management API calls for more.) With every authenticated request, the server returns a session cookie, which can be subsequently used for authentication."

curl -X GET -H "Content-Type:application/json" -u [username]:[password] \
-d '{ "items": [
{"name": "enable_config_alerts", "value": "true"} ] }' \
'http://[cm_address]:7180/api/v33/clusters/Cluster%201/services/KAFKA-1/clientConfig'

Please let me know if it works for you too!

Thank you:
Ferenc

[#7]

@Bender Yes, if specifying the username and password while accessing this API via curl or wget, it can get 'clientConfig' successfully. But the specification of Cloudera Manager API said this API does not require any authentication. Ref: https://archive.cloudera.com/cm6/6.3.0/generic/jar/cm_api/apidocs/resource_ServicesResource.html#res...

There is a little background for your reference, I have a program, while it's started, it will call 'wget http://<cm_server_host>:7180/api/v13/clusters/{clusterName}/services/{serviceName}/clientConfig'  directly (without specifying username/password) to get config file, and it works well with Cloudera Manager 5.10.0. After i upgraded Cloudera Manager to 6.3.0, wget will get a 404 error, then the program failed to be started.

[#8 · Accepted solution]

Hello @iEason8 ,

you've got a good point. The main page for the doc (which I've cited earlier) says you need authentication for API calls for CM6 however, the section you've found says explicitly it does not require authentication.

So I've done some research in our records and found that the desired behaviour for CM6 is to enforce authentication for API calls (internal jira reference OPSAPS-44459) and that there is a bug in our doc (internal jira reference DOCS-4659) that explains "Doc wrongly suggests that clientConfig API call does not require any authentication" and this jira is unresolved at this moment.

Sorry for the inconvenience caused. You need to get authenticated to be able to use the API in CM6.

Kind regards:

Ferenc

[#9]

@Bender Got it. Thanks so much for your clarification.

[#10]

@Bender One more minor thing, as I mentioned before, the error message is "Service 'yarn' not found in cluster 'platform-pod10-eason-cdh'" and error number is 404 while access this API without specifying credential, which is confusing. I think it is supposed to return 401 with authentication error message. Can you help to confirm it? Thanks!

[#11]

Hello @iEason8 ,

Thank you for your observation on the API returns 404 when a request was sent to CM API without authentication.

Based on W3C recommendation about authentication:

"... If the server does not wish to make this information available to the client, the status code 404 (Not Found) can be used instead."

Therefore it is OK to return 404 when there is an unauthenticated request and it is a design decision.

Kind regards:

Ferenc