Support Questions
Find answers, ask questions, and share your expertise

Metron 0.7.2 upgrade with ElasticSearch 6.X

Hello,

I have installed Apache Metron in a cluster created with Ambari 2.6.2.2 and HDP 2.6.5.1175, ES (Elasticsearch) 5.6.14 and K (Kibana) 5.6.14.

I followed this guides:

Later, I tried to upgrade ES & K to 6.X, knowing about the breaking changes.

I followed this guide and all went as expected. I reindexed and changed the templates so they are compliant with new changes of 6.X version, but the problem is:

 

I am not able to index anything to ES when I use the kafka queue.

 

It was working when 5.X, and ES is working great alone. I guess the problem is due to strict content-type checking introduced in ElasticSearch 6.0, as explained in this post, but I do not know how to include the 'Content-Type: application/json' header in the ES writing bolt in the STORM topology. Anyone knows what file I should change?

 

My environment specs are:

  • Java JDK 1.8 in all hosts
  • Metron RPM packages in /localrepo in all hosts
  • root user in all hosts
  • Ambari server 2.7.5.0
  • 6 Centos 7 host :
    • metron1 (9.6 GB, 4 cores, 100G SSD)
      • NameNode ZooKeeper Server Kafka Broker Zeppelin Notebook
    • metron2 (9.6 GB, 4 cores, 100G SSD)
      • SNameNode App Timeline Server ResourceManager History Server ZooKeeper Server Nimbus Kafka Broker
    • metron3 (15.5 GB, 4 cores, 500G SSD)
      • HBase Master Kafka Broker Elasticsearch Master
    • metron4 (11.6 GB, 3 cores, 50G SSD)
      • HBase Master Oozie Server Nimbus DRPC Server Storm UI Server Kafka Broker
    • metron5 (11.6 GB, 3 cores, 50G SSD)
      • Kafka Broker
    • metron6 (15.5 GB, 6 cores, 100G SSD)
      • HBase Master ZooKeeper Server Kafka Broker Kibana Server
0 REPLIES 0