Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Metron 0.7.2 upgrade with ElasticSearch 6.X

Metron 0.7.2 upgrade with ElasticSearch 6.X

Hello,

I have installed Apache Metron in a cluster created with Ambari 2.6.2.2 and HDP 2.6.5.1175, ES (Elasticsearch) 5.6.14 and K (Kibana) 5.6.14.

I followed this guides:

Later, I tried to upgrade ES & K to 6.X, knowing about the breaking changes.

I followed this guide and all went as expected. I reindexed and changed the templates so they are compliant with new changes of 6.X version, but the problem is:

 

I am not able to index anything to ES when I use the kafka queue.

 

It was working when 5.X, and ES is working great alone. I guess the problem is due to strict content-type checking introduced in ElasticSearch 6.0, as explained in this post, but I do not know how to include the 'Content-Type: application/json' header in the ES writing bolt in the STORM topology. Anyone knows what file I should change?

 

My environment specs are:

  • Java JDK 1.8 in all hosts
  • Metron RPM packages in /localrepo in all hosts
  • root user in all hosts
  • Ambari server 2.7.5.0
  • 6 Centos 7 host :
    • metron1 (9.6 GB, 4 cores, 100G SSD)
      • NameNode ZooKeeper Server Kafka Broker Zeppelin Notebook
    • metron2 (9.6 GB, 4 cores, 100G SSD)
      • SNameNode App Timeline Server ResourceManager History Server ZooKeeper Server Nimbus Kafka Broker
    • metron3 (15.5 GB, 4 cores, 500G SSD)
      • HBase Master Kafka Broker Elasticsearch Master
    • metron4 (11.6 GB, 3 cores, 50G SSD)
      • HBase Master Oozie Server Nimbus DRPC Server Storm UI Server Kafka Broker
    • metron5 (11.6 GB, 3 cores, 50G SSD)
      • Kafka Broker
    • metron6 (15.5 GB, 6 cores, 100G SSD)
      • HBase Master ZooKeeper Server Kafka Broker Kibana Server
Don't have an account?
Coming from Hortonworks? Activate your account here