Created 11-09-2017 01:32 AM
Hi,
I just installed Cloudera Manager 5.13 and "Parcel" installed the "all services" cluster. I haven't run anything on the cluster yet. But the Impala goes down and give me this:
"Bad : The Cloudera Manager Agent got an unexpected response from this role's web server."
As details, it says:
"The Cloudera Manager Agent got an unexpected response from this role's web server."
and
"The health test result for CATALOGSERVER_WEB_METRIC_COLLECTION has become bad: The Cloudera Manager Agent got an unexpected response from this role's web server."
It seems meaning that my catalog server send sth to the agent which was not understandable. So I checked the log of my server and two daemons. (I restarted the service at 9:46. The web server issue happened on 9:48:34)
Impala Catalog Server:
9:46:38.169 AM | INFO | cc:110 | CatalogService started on port: 26000 |
9:46:40.110 AM | INFO | cc:241 | Catalog Version: 3 Last Catalog Version: 0 |
9:47:03.068 AM | INFO | cc:296 | Old log file deleted during log rotation: /var/log/catalogd/catalogd.master.impala.log.INFO.20171108-214603.2421 |
9:47:03.068 AM | INFO | cc:296 | Old log file deleted during log rotation: /var/log/catalogd/catalogd.master.impala.log.WARNING.20171108-214603.2421 |
9:47:03.068 AM | INFO | cc:296 | Old log file deleted during log rotation: /var/log/catalogd/catalogd.master.impala.log.ERROR.20171108-214603.2421 |
9:56:40.166 AM | INFO | cc:241 | Catalog Version: 3 Last Catalog Version: 3 |
10:06:40.219 AM | INFO | cc:241 | Catalog Version: 3 Last Catalog Version: 3 |
Impala Daemons (these two are pretty same so I just post one of them here):
9:46:33.795 AM | INFO | cc:113 | Impala has started. |
9:47:01.823 AM | INFO | cc:296 | Old log file deleted during log rotation: /var/log/impalad/impalad.slave02.impala.log.INFO.20171108-214602.18311 |
9:47:01.824 AM | INFO | cc:296 | Old log file deleted during log rotation: /var/log/impalad/impalad.slave02.impala.log.WARNING.20171108-214602.18311 |
9:47:01.824 AM | INFO | cc:296 | Old log file deleted during log rotation: /var/log/impalad/impalad.slave02.impala.log.ERROR.20171108-214602.18311 |
But I found nothing goes wrong, so as stacks logs.
Then I tried to restart the service and the whole cluster. But these operations do not work. The issue always appeares soon after my each restart.
Could anyone tell me what's the problem and how to resolve it? Thanks!
Created 12-28-2017 03:00 AM
Hello !
I'm facing the same issue, can you post what you have found if y've already been fixing this issue ?
Many thanks
Created 01-08-2018 09:09 AM
Created 01-07-2018 04:08 PM
The health check that fails is run by the Cloudera Manager Agent.
On the host where the problem occurs, review your agent log file (/var/log/cloudera-scm-agent/cloudera-scm-agent.log by default) and check to see what sort of errors or stack trace is displayed when the agent attempts to connect to the server (Catalog Server in this case).
That should give us more information to review.
Created 01-07-2018 07:06 PM
Hello aisp,
This is an issue with your cloudera agent on the impacted host. While you are checking the logs of the agent, I would request you to look at the CM: All hosts -> Status column value.
Created 01-29-2018 07:52 AM
For us the problem was a "wrong" locale on the server (env LANG). That meant that the generated JSON was using commas as decimal delimiter instead of dots (e.g. "1,234" instead of "1.234").
We had to change the Impala Environment Safety Valve to include LC_ALL=C
Created 11-28-2018 10:17 AM
This did the trick for us, too.
Error messages
in cloudera-scm-agent.log where leading us to this configuration problem.
Thanks for sharing your fix!