Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

HUE - ERROR in handle_mutual_auth

avatar

Hi,

 noticed in Hue logs the following error:

 

[11/Sep/2018 13:35:59 +0200] kerberos_    ERROR    handle_mutual_auth(): Mutual authentication unavailable on 404 response

 

The Hue service works mostly as expected but have some times failures, not logging any usefull information (except this)

Is this error related to some bad configuration of the Hue services (two running in HA mode)?

Thanks

6 REPLIES 6

avatar
Master Guru

@Tomas79,

 

That error isn't really an error I think.  Since there is an HTTP 404 response, there is no use in doing mutual auth (kerberos).

Can you take some scren shots of the failures or show how you know there are failures?

Also, please include any logging (even INFO level) from the time when the error occurs (/var/log/hue/runcpserver.log)

 

Cheers!

avatar
I will do in the next Hue failure.

avatar

Today I catched another Hue failure.

hue-fail.PNG

Here is the content of runcpserver.log:

[18/Sep/2018 15:04:52 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:04:52 +0200] access       INFO     212.45.173.106, 10.197.144.123 wth8400 - "POST /jobbrowser/api/jobs HTTP/1.1"
[18/Sep/2018 15:04:52 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:04:52 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:05:04 +0200] access       INFO     212.45.173.105, 10.197.144.123 eyl7443 - "POST /jobbrowser/jobs/ HTTP/1.1"
[18/Sep/2018 15:05:04 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:05:04 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:05:04 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:05:04 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:05:17 +0200] access       INFO     212.45.173.106, 10.197.144.123 wth8400 - "POST /jobbrowser/jobs/ HTTP/1.1"
[18/Sep/2018 15:05:17 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:05:17 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:05:17 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:05:17 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:05:18 +0200] access       INFO     212.45.173.106, 10.197.144.123 wth8400 - "POST /jobbrowser/api/jobs HTTP/1.1"
[18/Sep/2018 15:05:18 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:05:18 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:05:18 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:05:18 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:05:19 +0200] access       INFO     212.45.173.106, 10.197.144.123 wth8400 - "POST /jobbrowser/api/jobs HTTP/1.1"
[18/Sep/2018 15:05:19 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:05:19 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:05:19 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:05:19 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:05:43 +0200] access       INFO     212.45.173.105, 10.197.144.123 eyl7443 - "POST /jobbrowser/jobs/ HTTP/1.1"
[18/Sep/2018 15:05:43 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:05:43 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:05:43 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:05:43 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:05:51 +0200] access       INFO     212.45.173.106, 10.197.144.123 wth8400 - "POST /jobbrowser/api/jobs HTTP/1.1"
[18/Sep/2018 15:05:51 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:05:51 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:05:51 +0200] access       INFO     212.45.173.106, 10.197.144.123 wth8400 - "POST /jobbrowser/api/jobs HTTP/1.1"
[18/Sep/2018 15:05:51 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:05:51 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:06:04 +0200] access       INFO     212.45.173.106, 10.197.144.123 wth8400 - "POST /jobbrowser/jobs/ HTTP/1.1"
[18/Sep/2018 15:06:04 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:06:04 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:06:04 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:06:04 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:06:17 +0200] decorators   INFO     AXES: Calling decorated function: dt_login
[18/Sep/2018 15:06:20 +0200] access       INFO     212.45.173.105, 10.197.144.123 eyl7443 - "POST /jobbrowser/jobs/ HTTP/1.1"
[18/Sep/2018 15:06:20 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:06:20 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:06:20 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:06:20 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:06:21 +0200] access       INFO     212.45.173.106, 10.197.144.123 wth8400 - "POST /jobbrowser/api/jobs HTTP/1.1"
[18/Sep/2018 15:06:21 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:06:21 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:06:21 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:06:21 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:06:21 +0200] access       INFO     212.45.173.106, 10.197.144.123 wth8400 - "POST /jobbrowser/api/jobs HTTP/1.1"
[18/Sep/2018 15:06:21 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:06:21 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:06:21 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:06:21 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:06:45 +0200] access       INFO     212.45.173.106, 10.197.144.123 wth8400 - "POST /jobbrowser/api/jobs HTTP/1.1"
[18/Sep/2018 15:06:45 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:06:45 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:06:45 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:06:45 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:06:46 +0200] access       INFO     212.45.173.106, 10.197.144.123 wth8400 - "POST /jobbrowser/api/jobs HTTP/1.1"
[18/Sep/2018 15:06:46 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:06:46 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:06:46 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:06:46 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:06:47 +0200] decorators   INFO     AXES: Calling decorated function: dt_login
[18/Sep/2018 15:06:54 +0200] access       INFO     212.45.173.105, 10.197.144.123 eyl7443 - "POST /jobbrowser/jobs/ HTTP/1.1"
[18/Sep/2018 15:06:54 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:06:54 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:06:54 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:06:54 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:06:56 +0200] access       INFO     212.45.173.106, 10.197.144.123 wth8400 - "POST /jobbrowser/jobs/ HTTP/1.1"
[18/Sep/2018 15:06:56 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:06:56 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:06:56 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:06:56 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:07:04 +0200] access       INFO     212.45.173.106, 10.197.144.123 wth8400 - "POST /jobbrowser/api/jobs HTTP/1.1"
[18/Sep/2018 15:07:04 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:07:04 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:07:04 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:07:04 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:07:04 +0200] access       INFO     212.45.173.106, 10.197.144.123 wth8400 - "POST /jobbrowser/api/jobs HTTP/1.1"
[18/Sep/2018 15:07:04 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:07:04 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:07:04 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:07:04 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:07:04 +0200] access       INFO     212.45.173.105, 10.197.132.245 nfc1092 - "POST /notebook/api/check_status HTTP/1.1"
[18/Sep/2018 15:07:05 +0200] access       INFO     212.45.173.105, 10.197.132.245 nfc1092 - "POST /notebook/api/check_status HTTP/1.1"
[18/Sep/2018 15:07:06 +0200] access       INFO     212.45.173.105, 10.197.132.245 nfc1092 - "POST /notebook/api/check_status HTTP/1.1"
[18/Sep/2018 15:07:22 +0200] access       INFO     212.45.173.106, 10.197.144.123 wth8400 - "POST /jobbrowser/api/jobs HTTP/1.1"
[18/Sep/2018 15:07:22 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:07:22 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:07:22 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:07:22 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:07:22 +0200] access       INFO     212.45.173.106, 10.197.144.123 wth8400 - "POST /jobbrowser/api/jobs HTTP/1.1"
[18/Sep/2018 15:07:22 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:07:22 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:07:23 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:07:23 +0200] cluster      WARNING  Picking RM HA: ha


No error..

The last log is from 15:07

 

avatar
Master Guru

@Tomas79,

 

Thanks, that clears things up a lot.

 

Cloudera Manager does health checks and metrics collection via the Hue UI.

The Cloudera Manager Agent that runs on the host where the Hue process is will periodically connect to your Hue instance and request "desktop/debug/is_alive".

 

If the agent does not get a response quickly enough, you can see Cloudera Manager report the bad health.

 

I don't see any "is_alive" requests in your logging.

If there was a failure, the agent log should show it.  Check your /var/log/cloudera-scm-agent/cloudera-scm-agent.log on the Hue host to see if you can find any failures pertaining to "is_alive" or hue in general.

 

What version of CDH are you on, by the way?

avatar
I will check it. CDH 5.13
Do I understand your explanation correctly that this is a problem that the
agent does not "see" the hue service alivve? But then it would just report
a false alarm, but actually the hue server was not available, I did not
investigated the processes but I am quite sure that the Hue process was not
running

avatar
Master Guru

@Tomas79,

 

If Hue is shut down from Cloudera Manager, those health checks should cease automatically.

If Hue exited unexpectedly you may get these alerts

 

The agent log will tell us much more about the story since it will show what it knew of the Hue process from a start/stop perspective and also it will show any issues getting results from the Hue UI.