Metron "metron_elasticsearch_templates : Wait for Index to Become Available" failure on physical deployment

zeolla — Wed, 07 Sep 2016 09:35:59 GMT

I'm running Metron on an 8 node physical cluster (CentOS 6), with a virtual installer node (CentOS 7).

I'm running into the following error after running `ansible-playbook -i /root/incubator-metron/metron-deployment/inventory/$INSTALL_CONF_DIRECTORY metron_install.yml --skip-tags="solr,sensors"`:

TASK [metron_elasticsearch_templates : Wait for Index to Become Available] *****
fatal: [server2]: FAILED! => {"failed": true, "msg": "ERROR! The conditional check 'result.content.find(\"green\") != -1 or result.content.find(\"yellow\") != -1' failed. The error was: ERROR! error while evaluating conditional (result.content.find(\"green\") != -1 or result.content.find(\"yellow\") != -1): ERROR! 'dict object' has no attribute 'content'"}
fatal: [server2]: FAILED! => {"failed": true, "msg": "ERROR! The conditional check 'result.content.find(\"green\") != -1 or result.content.find(\"yellow\") != -1' failed. The error was: ERROR! error while evaluating conditional (result.content.find(\"green\") != -1 or result.content.find(\"yellow\") != -1): ERROR! 'dict object' has no attribute 'content'"}

However when I check the servers manually I see:

[root@e104d0xxxxxx scripts]# curl server1:9200/_cluster/health
{"cluster_name":"metron","status":"green","timed_out":false,"number_of_nodes":2,"number_of_data_nodes":2,"active_primary_shards":0,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":100.0}[root@e104d0553817 scripts]#
[root@e104d0xxxxxx scripts]# curl server2:9200/_cluster/health
{"cluster_name":"metron","status":"green","timed_out":false,"number_of_nodes":2,"number_of_data_nodes":2,"active_primary_shards":0,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":100.0}[root@e104d0553817 scripts]#

And when I check in my web browser, I get:

1473215295 22:28:15 metron green 2 2 0 0 0 0 0 0 - 100.0%
1473215299 22:28:19 metron green 2 2 0 0 0 0 0 0 - 100.0%

I've tried editing incubator-metron/metron-deployment/roles/metron_elasticsearch_templates/tasks/load_templates.yml and increase the retries (up to 600) and delays (up to 1000) with no success.

Some details:

Metron 0.2.0BETA
--
* (no branch)
--
commit 75642001803396e8884385b0fc297a2312ead3eb
Author: cstella <cestella@gmail.com>
Date: Wed Jul 13 11:15:20 2016 -0400
METRON-298 Remove the effective_tld_names.dat files. closes apache/incubator-metron#186
--
metron-deployment/playbooks/metron_install.yml | 45 ++++++++++++++++++++
.../roles/ambari_common/tasks/main.yml | 5 ++-
.../roles/elasticsearch/defaults/main.yml | 2 +-
.../roles/elasticsearch/tasks/elasticsearch.yml | 2 +-
metron-deployment/roles/httplib2/tasks/main.yml | 3 +-
.../roles/kibana/tasks/elasticdump.yml | 4 +-
.../roles/opentaxii/tasks/opentaxii.yml | 2 +-
7 files changed, 57 insertions(+), 6 deletions(-)
--
ansible 2.0.0.2
config file = /root/ansible.cfg
configured module search path = ../extra_modules
--
./platform-info.sh: line 52: vagrant: command not found
--
Python 2.6.6
--
Apache Maven 3.2.5 (12a6b3acb947671f09b81f49094c53f426d8cea1; 2014-12-14T17:29:23+00:00)
Maven home: /opt/maven
Java version: 1.8.0_91, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.91-1.b14.el6.x86_64/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-327.el7.x86_64", arch: "amd64", family: "unix"
--
Linux e104d0553817 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Re: Metron "metron_elasticsearch_templates : Wait for Index to Become Available" failure on physical deployment

zeolla — Wed, 07 Sep 2016 20:01:32 GMT

The issue was because I had set environment variables to use our proxies and it was trying to proxy the cluster health check, but when I did it at the command line that env var wasn't set. I set no_proxy for server1,server2 and it worked properly.

question Re: Metron "metron_elasticsearch_templates : Wait for Index to Become Available" failure on physical deployment in Archives of Support Questions (Read Only)

Metron "metron_elasticsearch_templates : Wait for Index to Become Available" failure on physical deployment

Re: Metron "metron_elasticsearch_templates : Wait for Index to Become Available" failure on physical deployment