Impala Catalog Server gets Web Server Status Issue after each restart

[#1]

Hi,

I just installed Cloudera Manager 5.13 and "Parcel" installed the "all services" cluster. I haven't run anything on the cluster yet. But the Impala goes down and give me this:

"Bad : The Cloudera Manager Agent got an unexpected response from this role's web server."

As details, it says:

"The Cloudera Manager Agent got an unexpected response from this role's web server."

and

"The health test result for CATALOGSERVER_WEB_METRIC_COLLECTION has become bad: The Cloudera Manager Agent got an unexpected response from this role's web server."

It seems meaning that my catalog server send sth to the agent which was not understandable. So I checked the log of my server and two daemons. (I restarted the service at 9:46. The web server issue happened on 9:48:34)

Impala Catalog Server:

9:46:38.169 AM	INFO	cc:110	CatalogService started on port: 26000
9:46:40.110 AM	INFO	cc:241	Catalog Version: 3 Last Catalog Version: 0
9:47:03.068 AM	INFO	cc:296	Old log file deleted during log rotation: /var/log/catalogd/catalogd.master.impala.log.INFO.20171108-214603.2421
9:47:03.068 AM	INFO	cc:296	Old log file deleted during log rotation: /var/log/catalogd/catalogd.master.impala.log.WARNING.20171108-214603.2421
9:47:03.068 AM	INFO	cc:296	Old log file deleted during log rotation: /var/log/catalogd/catalogd.master.impala.log.ERROR.20171108-214603.2421
9:56:40.166 AM	INFO	cc:241	Catalog Version: 3 Last Catalog Version: 3
10:06:40.219 AM	INFO	cc:241	Catalog Version: 3 Last Catalog Version: 3

Impala Daemons (these two are pretty same so I just post one of them here):

9:46:33.795 AM	INFO	cc:113	Impala has started.
9:47:01.823 AM	INFO	cc:296	Old log file deleted during log rotation: /var/log/impalad/impalad.slave02.impala.log.INFO.20171108-214602.18311
9:47:01.824 AM	INFO	cc:296	Old log file deleted during log rotation: /var/log/impalad/impalad.slave02.impala.log.WARNING.20171108-214602.18311
9:47:01.824 AM	INFO	cc:296	Old log file deleted during log rotation: /var/log/impalad/impalad.slave02.impala.log.ERROR.20171108-214602.18311

But I found nothing goes wrong, so as stacks logs.

Then I tried to restart the service and the whole cluster. But these operations do not work. The issue always appeares soon after my each restart.

Could anyone tell me what's the problem and how to resolve it? Thanks!

[#2]

Hello !

I'm facing the same issue, can you post what you have found if y've already been fixing this issue ?

Many thanks

[#3]

Hi there,
Unfortunately I have no solution til now. I have to block Impala at last. Maybe you can check others' replies to get some idea.

[#4]

@aispand @skhadrao,

The health check that fails is run by the Cloudera Manager Agent.

On the host where the problem occurs, review your agent log file (/var/log/cloudera-scm-agent/cloudera-scm-agent.log by default) and check to see what sort of errors or stack trace is displayed when the agent attempts to connect to the server (Catalog Server in this case).

That should give us more information to review.

[#5]

Hello aisp,

This is an issue with your cloudera agent on the impacted host. While you are checking the logs of the agent, I would request you to look at the CM: All hosts -> Status column value.

[#6]

For us the problem was a "wrong" locale on the server (env LANG). That meant that the generated JSON was using commas as decimal delimiter instead of dots (e.g. "1,234" instead of "1.234").

We had to change the Impala Environment Safety Valve to include LC_ALL=C

[#7]

This did the trick for us, too.

Error messages

1. "ImpalaDaemonQueryMonitoring throttling_logger ERROR" ...
2. "ERROR fetching metrics at 'http:/xxx/jsonmetrics?json' " and
3. "JSONDecodingError: Expecting property name enclosed in double quotes: line 85 column 19"

in cloudera-scm-agent.log where leading us to this configuration problem.

Thanks for sharing your fix!