Reply
Master
Posts: 304
Registered: ‎07-01-2015

HUE - ERROR in handle_mutual_auth

[ Edited ]

Hi,

 noticed in Hue logs the following error:

 

[11/Sep/2018 13:35:59 +0200] kerberos_    ERROR    handle_mutual_auth(): Mutual authentication unavailable on 404 response

 

The Hue service works mostly as expected but have some times failures, not logging any usefull information (except this)

Is this error related to some bad configuration of the Hue services (two running in HA mode)?

Thanks

Posts: 911
Topics: 1
Kudos: 208
Solutions: 111
Registered: ‎04-22-2014

Re: HUE - ERROR in handle_mutual_auth

@Tomas79,

 

That error isn't really an error I think.  Since there is an HTTP 404 response, there is no use in doing mutual auth (kerberos).

Can you take some scren shots of the failures or show how you know there are failures?

Also, please include any logging (even INFO level) from the time when the error occurs (/var/log/hue/runcpserver.log)

 

Cheers!

Master
Posts: 304
Registered: ‎07-01-2015

Re: HUE - ERROR in handle_mutual_auth

I will do in the next Hue failure.
Master
Posts: 304
Registered: ‎07-01-2015

Re: HUE - ERROR in handle_mutual_auth

Today I catched another Hue failure.

hue-fail.PNG

Here is the content of runcpserver.log:

[18/Sep/2018 15:04:52 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:04:52 +0200] access       INFO     212.45.173.106, 10.197.144.123 wth8400 - "POST /jobbrowser/api/jobs HTTP/1.1"
[18/Sep/2018 15:04:52 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:04:52 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:05:04 +0200] access       INFO     212.45.173.105, 10.197.144.123 eyl7443 - "POST /jobbrowser/jobs/ HTTP/1.1"
[18/Sep/2018 15:05:04 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:05:04 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:05:04 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:05:04 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:05:17 +0200] access       INFO     212.45.173.106, 10.197.144.123 wth8400 - "POST /jobbrowser/jobs/ HTTP/1.1"
[18/Sep/2018 15:05:17 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:05:17 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:05:17 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:05:17 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:05:18 +0200] access       INFO     212.45.173.106, 10.197.144.123 wth8400 - "POST /jobbrowser/api/jobs HTTP/1.1"
[18/Sep/2018 15:05:18 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:05:18 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:05:18 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:05:18 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:05:19 +0200] access       INFO     212.45.173.106, 10.197.144.123 wth8400 - "POST /jobbrowser/api/jobs HTTP/1.1"
[18/Sep/2018 15:05:19 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:05:19 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:05:19 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:05:19 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:05:43 +0200] access       INFO     212.45.173.105, 10.197.144.123 eyl7443 - "POST /jobbrowser/jobs/ HTTP/1.1"
[18/Sep/2018 15:05:43 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:05:43 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:05:43 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:05:43 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:05:51 +0200] access       INFO     212.45.173.106, 10.197.144.123 wth8400 - "POST /jobbrowser/api/jobs HTTP/1.1"
[18/Sep/2018 15:05:51 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:05:51 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:05:51 +0200] access       INFO     212.45.173.106, 10.197.144.123 wth8400 - "POST /jobbrowser/api/jobs HTTP/1.1"
[18/Sep/2018 15:05:51 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:05:51 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:06:04 +0200] access       INFO     212.45.173.106, 10.197.144.123 wth8400 - "POST /jobbrowser/jobs/ HTTP/1.1"
[18/Sep/2018 15:06:04 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:06:04 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:06:04 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:06:04 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:06:17 +0200] decorators   INFO     AXES: Calling decorated function: dt_login
[18/Sep/2018 15:06:20 +0200] access       INFO     212.45.173.105, 10.197.144.123 eyl7443 - "POST /jobbrowser/jobs/ HTTP/1.1"
[18/Sep/2018 15:06:20 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:06:20 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:06:20 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:06:20 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:06:21 +0200] access       INFO     212.45.173.106, 10.197.144.123 wth8400 - "POST /jobbrowser/api/jobs HTTP/1.1"
[18/Sep/2018 15:06:21 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:06:21 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:06:21 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:06:21 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:06:21 +0200] access       INFO     212.45.173.106, 10.197.144.123 wth8400 - "POST /jobbrowser/api/jobs HTTP/1.1"
[18/Sep/2018 15:06:21 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:06:21 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:06:21 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:06:21 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:06:45 +0200] access       INFO     212.45.173.106, 10.197.144.123 wth8400 - "POST /jobbrowser/api/jobs HTTP/1.1"
[18/Sep/2018 15:06:45 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:06:45 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:06:45 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:06:45 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:06:46 +0200] access       INFO     212.45.173.106, 10.197.144.123 wth8400 - "POST /jobbrowser/api/jobs HTTP/1.1"
[18/Sep/2018 15:06:46 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:06:46 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:06:46 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:06:46 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:06:47 +0200] decorators   INFO     AXES: Calling decorated function: dt_login
[18/Sep/2018 15:06:54 +0200] access       INFO     212.45.173.105, 10.197.144.123 eyl7443 - "POST /jobbrowser/jobs/ HTTP/1.1"
[18/Sep/2018 15:06:54 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:06:54 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:06:54 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:06:54 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:06:56 +0200] access       INFO     212.45.173.106, 10.197.144.123 wth8400 - "POST /jobbrowser/jobs/ HTTP/1.1"
[18/Sep/2018 15:06:56 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:06:56 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:06:56 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:06:56 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:07:04 +0200] access       INFO     212.45.173.106, 10.197.144.123 wth8400 - "POST /jobbrowser/api/jobs HTTP/1.1"
[18/Sep/2018 15:07:04 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:07:04 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:07:04 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:07:04 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:07:04 +0200] access       INFO     212.45.173.106, 10.197.144.123 wth8400 - "POST /jobbrowser/api/jobs HTTP/1.1"
[18/Sep/2018 15:07:04 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:07:04 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:07:04 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:07:04 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:07:04 +0200] access       INFO     212.45.173.105, 10.197.132.245 nfc1092 - "POST /notebook/api/check_status HTTP/1.1"
[18/Sep/2018 15:07:05 +0200] access       INFO     212.45.173.105, 10.197.132.245 nfc1092 - "POST /notebook/api/check_status HTTP/1.1"
[18/Sep/2018 15:07:06 +0200] access       INFO     212.45.173.105, 10.197.132.245 nfc1092 - "POST /notebook/api/check_status HTTP/1.1"
[18/Sep/2018 15:07:22 +0200] access       INFO     212.45.173.106, 10.197.144.123 wth8400 - "POST /jobbrowser/api/jobs HTTP/1.1"
[18/Sep/2018 15:07:22 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:07:22 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:07:22 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:07:22 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:07:22 +0200] access       INFO     212.45.173.106, 10.197.144.123 wth8400 - "POST /jobbrowser/api/jobs HTTP/1.1"
[18/Sep/2018 15:07:22 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:07:22 +0200] cluster      WARNING  Picking RM HA: ha
[18/Sep/2018 15:07:23 +0200] cluster      INFO     RM default is not RUNNING, skipping it: {u'clusterInfo': {u'hadoopVersionBuiltOn': u'2018-03-17T11:29Z', u'startedOn': 1535656203104, u'haState': u'STANDBY', u'haZooKeeperConnectionState': u'CONNECTED', u'resourceManagerBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum c135844b5395ad0c0cdfd8ee671cccb', u'resourceManagerVersion': u'2.6.0-cdh5.13.3', u'state': u'STARTED', u'hadoopVersion': u'2.6.0-cdh5.13.3', u'resourceManagerVersionBuiltOn': u'2018-03-17T11:38Z', u'rmStateStoreName': u'org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore', u'hadoopBuildVersion': u'2.6.0-cdh5.13.3 from ad398b626d788a1fa5efbe790f38255540f31358 by jenkins source checksum 49734dccd1350506f3b98dff08f1438', u'id': 1535656203104}}
[18/Sep/2018 15:07:23 +0200] cluster      WARNING  Picking RM HA: ha


No error..

The last log is from 15:07

 

Posts: 911
Topics: 1
Kudos: 208
Solutions: 111
Registered: ‎04-22-2014

Re: HUE - ERROR in handle_mutual_auth

@Tomas79,

 

Thanks, that clears things up a lot.

 

Cloudera Manager does health checks and metrics collection via the Hue UI.

The Cloudera Manager Agent that runs on the host where the Hue process is will periodically connect to your Hue instance and request "desktop/debug/is_alive".

 

If the agent does not get a response quickly enough, you can see Cloudera Manager report the bad health.

 

I don't see any "is_alive" requests in your logging.

If there was a failure, the agent log should show it.  Check your /var/log/cloudera-scm-agent/cloudera-scm-agent.log on the Hue host to see if you can find any failures pertaining to "is_alive" or hue in general.

 

What version of CDH are you on, by the way?

Master
Posts: 304
Registered: ‎07-01-2015

Re: HUE - ERROR in handle_mutual_auth

I will check it. CDH 5.13
Do I understand your explanation correctly that this is a problem that the
agent does not "see" the hue service alivve? But then it would just report
a false alarm, but actually the hue server was not available, I did not
investigated the processes but I am quite sure that the Hue process was not
running
Posts: 911
Topics: 1
Kudos: 208
Solutions: 111
Registered: ‎04-22-2014

Re: HUE - ERROR in handle_mutual_auth

@Tomas79,

 

If Hue is shut down from Cloudera Manager, those health checks should cease automatically.

If Hue exited unexpectedly you may get these alerts

 

The agent log will tell us much more about the story since it will show what it knew of the Hue process from a start/stop perspective and also it will show any issues getting results from the Hue UI.

Announcements