Cloudera Community

Support Questions

Find answers, ask questions, and share your expertise
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search

HUE - ERROR in handle_mutual_auth

Labels:
avatar
author-rank Tomas79
Guru

Created on ‎09-11-2018 04:46 AM - edited ‎09-16-2022 06:41 AM

Hi,

 noticed in Hue logs the following error:

 

[11/Sep/2018 13:35:59 +0200] kerberos_    ERROR    handle_mutual_auth(): Mutual authentication unavailable on 404 response

 

The Hue service works mostly as expected but have some times failures, not logging any usefull information (except this)

Is this error related to some bad configuration of the Hue services (two running in HA mode)?

Thanks

4,502 Views
0 Kudos
6 REPLIES 6

avatar
author-rank bgooley author-rank
Master Guru

Created ‎09-17-2018 04:22 PM

@Tomas79,

 

That error isn't really an error I think.  Since there is an HTTP 404 response, there is no use in doing mutual auth (kerberos).

Can you take some scren shots of the failures or show how you know there are failures?

Also, please include any logging (even INFO level) from the time when the error occurs (/var/log/hue/runcpserver.log)

 

Cheers!

4,420 Views
0 Kudos

avatar
author-rank Tomas79
Guru

Created ‎09-17-2018 10:16 PM

I will do in the next Hue failure.
4,408 Views
0 Kudos

avatar
author-rank Tomas79
Guru

Created ‎09-18-2018 06:27 AM

Today I catched another Hue failure.

hue-fail.PNG

Here is the content of runcpserver.log:

[18/Sep/2018 15:04:52 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:04:52 +0200] access       INFO     212.45.173.106, 10.197.144.123 wth8400 - "POST /jobbrowser/api/jobs HTTP/1.1"
[18/Sep/2018 15:04:52 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:04:52 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:05:04 +0200] access       INFO     212.45.173.105, 10.197.144.123 eyl7443 - "POST /jobbrowser/jobs/ HTTP/1.1"
[18/Sep/2018 15:05:04 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:05:04 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:05:04 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:05:04 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:05:17 +0200] access       INFO     212.45.173.106, 10.197.144.123 wth8400 - "POST /jobbrowser/jobs/ HTTP/1.1"
[18/Sep/2018 15:05:17 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:05:17 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:05:17 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:05:17 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:05:18 +0200] access       INFO     212.45.173.106, 10.197.144.123 wth8400 - "POST /jobbrowser/api/jobs HTTP/1.1"
[18/Sep/2018 15:05:18 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:05:18 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:05:18 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:05:18 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:05:19 +0200] access       INFO     212.45.173.106, 10.197.144.123 wth8400 - "POST /jobbrowser/api/jobs HTTP/1.1"
[18/Sep/2018 15:05:19 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:05:19 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:05:19 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:05:19 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:05:43 +0200] access       INFO     212.45.173.105, 10.197.144.123 eyl7443 - "POST /jobbrowser/jobs/ HTTP/1.1"
[18/Sep/2018 15:05:43 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:05:43 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:05:43 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:05:43 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:05:51 +0200] access       INFO     212.45.173.106, 10.197.144.123 wth8400 - "POST /jobbrowser/api/jobs HTTP/1.1"
[18/Sep/2018 15:05:51 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:05:51 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:05:51 +0200] access       INFO     212.45.173.106, 10.197.144.123 wth8400 - "POST /jobbrowser/api/jobs HTTP/1.1"
[18/Sep/2018 15:05:51 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:05:51 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:06:04 +0200] access       INFO     212.45.173.106, 10.197.144.123 wth8400 - "POST /jobbrowser/jobs/ HTTP/1.1"
[18/Sep/2018 15:06:04 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:06:04 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:06:04 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:06:04 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:06:17 +0200] decorators   INFO     AXES: Calling decorated function: dt_login
[18/Sep/2018 15:06:20 +0200] access       INFO     212.45.173.105, 10.197.144.123 eyl7443 - "POST /jobbrowser/jobs/ HTTP/1.1"
[18/Sep/2018 15:06:20 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:06:20 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:06:20 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:06:20 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:06:21 +0200] access       INFO     212.45.173.106, 10.197.144.123 wth8400 - "POST /jobbrowser/api/jobs HTTP/1.1"
[18/Sep/2018 15:06:21 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:06:21 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:06:21 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:06:21 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:06:21 +0200] access       INFO     212.45.173.106, 10.197.144.123 wth8400 - "POST /jobbrowser/api/jobs HTTP/1.1"
[18/Sep/2018 15:06:21 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:06:21 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:06:21 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:06:21 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:06:45 +0200] access       INFO     212.45.173.106, 10.197.144.123 wth8400 - "POST /jobbrowser/api/jobs HTTP/1.1"
[18/Sep/2018 15:06:45 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:06:45 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:06:45 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:06:45 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:06:46 +0200] access       INFO     212.45.173.106, 10.197.144.123 wth8400 - "POST /jobbrowser/api/jobs HTTP/1.1"
[18/Sep/2018 15:06:46 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:06:46 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:06:46 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:06:46 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:06:47 +0200] decorators   INFO     AXES: Calling decorated function: dt_login
[18/Sep/2018 15:06:54 +0200] access       INFO     212.45.173.105, 10.197.144.123 eyl7443 - "POST /jobbrowser/jobs/ HTTP/1.1"
[18/Sep/2018 15:06:54 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:06:54 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:06:54 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:06:54 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:06:56 +0200] access       INFO     212.45.173.106, 10.197.144.123 wth8400 - "POST /jobbrowser/jobs/ HTTP/1.1"
[18/Sep/2018 15:06:56 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:06:56 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:06:56 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:06:56 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:07:04 +0200] access       INFO     212.45.173.106, 10.197.144.123 wth8400 - "POST /jobbrowser/api/jobs HTTP/1.1"
[18/Sep/2018 15:07:04 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:07:04 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:07:04 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:07:04 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:07:04 +0200] access       INFO     212.45.173.106, 10.197.144.123 wth8400 - "POST /jobbrowser/api/jobs HTTP/1.1"
[18/Sep/2018 15:07:04 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:07:04 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:07:04 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:07:04 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:07:04 +0200] access       INFO     212.45.173.105, 10.197.132.245 nfc1092 - "POST /notebook/api/check_status HTTP/1.1"
[18/Sep/2018 15:07:05 +0200] access       INFO     212.45.173.105, 10.197.132.245 nfc1092 - "POST /notebook/api/check_status HTTP/1.1"
[18/Sep/2018 15:07:06 +0200] access       INFO     212.45.173.105, 10.197.132.245 nfc1092 - "POST /notebook/api/check_status HTTP/1.1"
[18/Sep/2018 15:07:22 +0200] access       INFO     212.45.173.106, 10.197.144.123 wth8400 - "POST /jobbrowser/api/jobs HTTP/1.1"
[18/Sep/2018 15:07:22 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:07:22 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:07:22 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:07:22 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:07:22 +0200] access       INFO     212.45.173.106, 10.197.144.123 wth8400 - "POST /jobbrowser/api/jobs HTTP/1.1"
[18/Sep/2018 15:07:22 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:07:22 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:07:23 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:07:23 +0200] cluster      WARNING  Picking RM HA: ha

No error..

The last log is from 15:07

 

4,390 Views
0 Kudos

avatar
author-rank bgooley author-rank
Master Guru

Created ‎09-18-2018 11:54 AM

@Tomas79,

 

Thanks, that clears things up a lot.

 

Cloudera Manager does health checks and metrics collection via the Hue UI.

The Cloudera Manager Agent that runs on the host where the Hue process is will periodically connect to your Hue instance and request "desktop/debug/is_alive".

 

If the agent does not get a response quickly enough, you can see Cloudera Manager report the bad health.

 

I don't see any "is_alive" requests in your logging.

If there was a failure, the agent log should show it.  Check your /var/log/cloudera-scm-agent/cloudera-scm-agent.log on the Hue host to see if you can find any failures pertaining to "is_alive" or hue in general.

 

What version of CDH are you on, by the way?

4,373 Views
0 Kudos

avatar
author-rank Tomas79
Guru

Created ‎09-18-2018 11:59 AM

I will check it. CDH 5.13
Do I understand your explanation correctly that this is a problem that the
agent does not "see" the hue service alivve? But then it would just report
a false alarm, but actually the hue server was not available, I did not
investigated the processes but I am quite sure that the Hue process was not
running
4,370 Views
0 Kudos

avatar
author-rank bgooley author-rank
Master Guru

Created ‎09-18-2018 12:37 PM

@Tomas79,

 

If Hue is shut down from Cloudera Manager, those health checks should cease automatically.

If Hue exited unexpectedly you may get these alerts

 

The agent log will tell us much more about the story since it will show what it knew of the Hue process from a start/stop perspective and also it will show any issues getting results from the Hue UI.

4,365 Views
0 Kudos
Announcements
What's New @ Cloudera
[RELEASED] Cloudera Streams Messaging - Kubernetes Operator ...
What's New @ Cloudera
[RELEASED] Cloudera Streaming Analytics 1.14 for Cloudera Pu...
What's New @ Cloudera
Cloudera Data Engineering 1.23: Access Spark from Your Favor...
What's New @ Cloudera
HBase REST server scaling support is Generally Available
What's New @ Cloudera
New CLI option in the update-database command
View More Announcements