Reply
New Contributor
Posts: 5
Registered: ‎11-08-2017

Impala Catalog Server gets Web Server Status Issue after each restart

Hi,

 

I just installed Cloudera Manager 5.13 and "Parcel" installed the "all services" cluster. I haven't run anything on the cluster yet. But the Impala goes down and give me this:

 

"Bad : The Cloudera Manager Agent got an unexpected response from this role's web server."

 

As details, it says:

 

"The Cloudera Manager Agent got an unexpected response from this role's web server."

and

"The health test result for CATALOGSERVER_WEB_METRIC_COLLECTION has become bad: The Cloudera Manager Agent got an unexpected response from this role's web server."

 

It seems meaning that my catalog server send sth to the agent which was not understandable. So I checked the log of my server and two daemons. (I restarted the service at 9:46. The web server issue happened on 9:48:34)

 

Impala Catalog Server:

 

9:46:38.169 AMINFOcc:110
CatalogService started on port: 26000
9:46:40.110 AMINFOcc:241
Catalog Version: 3 Last Catalog Version: 0
9:47:03.068 AMINFOcc:296
Old log file deleted during log rotation: /var/log/catalogd/catalogd.master.impala.log.INFO.20171108-214603.2421
9:47:03.068 AMINFOcc:296
Old log file deleted during log rotation: /var/log/catalogd/catalogd.master.impala.log.WARNING.20171108-214603.2421
9:47:03.068 AMINFOcc:296
Old log file deleted during log rotation: /var/log/catalogd/catalogd.master.impala.log.ERROR.20171108-214603.2421
9:56:40.166 AMINFOcc:241
Catalog Version: 3 Last Catalog Version: 3
10:06:40.219 AMINFOcc:241
Catalog Version: 3 Last Catalog Version: 3

 

Impala Daemons (these two are pretty same so I just post one of them here):

9:46:33.795 AMINFOcc:113
Impala has started.
9:47:01.823 AMINFOcc:296
Old log file deleted during log rotation: /var/log/impalad/impalad.slave02.impala.log.INFO.20171108-214602.18311
9:47:01.824 AMINFOcc:296
Old log file deleted during log rotation: /var/log/impalad/impalad.slave02.impala.log.WARNING.20171108-214602.18311
9:47:01.824 AMINFOcc:296
Old log file deleted during log rotation: /var/log/impalad/impalad.slave02.impala.log.ERROR.20171108-214602.18311

 

But I found nothing goes wrong, so as stacks logs.

 

Then I tried to restart the service and the whole cluster. But these operations do not work. The issue always appeares soon after my each restart.

 

 

Could anyone tell me what's the problem and how to resolve it? Thanks!

 

Explorer
Posts: 6
Registered: ‎12-28-2017

Re: Impala Catalog Server gets Web Server Status Issue after each restart

Hello !

 

I'm facing the same issue, can you post what you have found if y've already been fixing this issue ?

 

Many thanks

Posts: 953
Topics: 1
Kudos: 226
Solutions: 121
Registered: ‎04-22-2014

Re: Impala Catalog Server gets Web Server Status Issue after each restart

@aispand @skhadrao,

 

The health check that fails is run by the Cloudera Manager Agent.

On the host where the problem occurs, review your agent log file (/var/log/cloudera-scm-agent/cloudera-scm-agent.log by default) and check to see what sort of errors or stack trace is displayed when the agent attempts to connect to the server (Catalog Server in this case).

 

That should give us more information to review.

Contributor
Posts: 52
Registered: ‎01-08-2016

Re: Impala Catalog Server gets Web Server Status Issue after each restart

Hello aisp,

 

This is an issue with your cloudera agent on the impacted host. While you are checking the logs of the agent, I would request you to look at the CM: All hosts -> Status column value.

New Contributor
Posts: 5
Registered: ‎11-08-2017

Re: Impala Catalog Server gets Web Server Status Issue after each restart

Hi there,
Unfortunately I have no solution til now. I have to block Impala at last. Maybe you can check others' replies to get some idea.
New Contributor
Posts: 5
Registered: ‎01-29-2018

Re: Impala Catalog Server gets Web Server Status Issue after each restart

For us the problem was a "wrong" locale on the server (env LANG). That meant that the generated JSON was using commas as decimal delimiter instead of dots (e.g. "1,234" instead of "1.234").

 

We had to change the Impala Environment Safety Valve to include LC_ALL=C

Highlighted
New Contributor
Posts: 2
Registered: ‎02-02-2018

Re: Impala Catalog Server gets Web Server Status Issue after each restart

This did the trick for us, too.

Error messages

 

  1. "ImpalaDaemonQueryMonitoring throttling_logger ERROR" ...
  2. "ERROR fetching metrics at 'http:/xxx/jsonmetrics?json' " and
  3. "JSONDecodingError: Expecting property name enclosed in double quotes: line 85 column 19"

in cloudera-scm-agent.log where leading us to this configuration problem.

Thanks for sharing your fix!

 

 

Announcements