Later, I tried to upgrade ES & K to 6.X, knowing about the breaking changes.
I followed this guide and all went as expected. I reindexed and changed the templates so they are compliant with new changes of 6.X version, but the problem is:
I am not able to index anything to ES when I use the kafka queue.
It was working when 5.X, and ES is working great alone. I guess the problem is due to strict content-type checking introduced in ElasticSearch 6.0, as explained in this post, but I do not know how to include the 'Content-Type: application/json' header in the ES writing bolt in the STORM topology. Anyone knows what file I should change?
My environment specs are:
Java JDK 1.8 in all hosts
Metron RPM packages in /localrepo in all hosts
root user in all hosts
Ambari server 188.8.131.52
6 Centos 7 host :
metron1 (9.6 GB, 4 cores, 100G SSD)
NameNode ZooKeeper Server Kafka Broker Zeppelin Notebook
metron2 (9.6 GB, 4 cores, 100G SSD)
SNameNode App Timeline Server ResourceManager History Server ZooKeeper Server Nimbus Kafka Broker
metron3 (15.5 GB, 4 cores, 500G SSD)
HBase Master Kafka Broker Elasticsearch Master
metron4 (11.6 GB, 3 cores, 50G SSD)
HBase Master Oozie Server Nimbus DRPC Server Storm UI Server Kafka Broker
metron5 (11.6 GB, 3 cores, 50G SSD)
metron6 (15.5 GB, 6 cores, 100G SSD)
HBase Master ZooKeeper Server Kafka Broker Kibana Server