Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Elasticsearch and HDFS function in Metron (Hortonworks Cybersecurity Platform)

Highlighted

Elasticsearch and HDFS function in Metron (Hortonworks Cybersecurity Platform)

New Contributor

Hi All,

I read about metron can use Elsticsearch and HDFS. It said that Elasticsearch is used for short term data retention for dashboard and analysis and HDFS for long term data retention archive.

is it possible to only use elasticsearch without HDFS?

Is there any process which make using HDFS important for HCP?

Thanks

3 REPLIES 3

Re: Elasticsearch and HDFS function in Metron (Hortonworks Cybersecurity Platform)

Super Collaborator

Hello @Rudy Hartono,

You can disable writing into HDFS per sensor by changing the 'enabled' indexing configuration under $METRON_HOME/config/zookeeper/indexing/<your-sensor>.json

For e.g., here's a sample for the bro sensor:

[metron@metron-1 ~]$ cat $METRON_HOME/config/zookeeper/indexing/bro.json
{
  "hdfs" : {
    "index": "bro",
    "batchSize": 5,
    "enabled" : true
  },
  "elasticsearch" : {
    "index": "bro",
    "batchSize": 5,
    "enabled" : true
  },
  "solr" : {
    "index": "bro",
    "batchSize": 5,
    "enabled" : true
  }
}

Note: Once you change the configuration, you need to use the 'zk_load_configs.sh' command to push the config changes into ZK. Refer here for more details.

As to your other question.. for metron, HDFS serves as an archival store for an analyst to refer to at a later stage. As of now, the UI interface does not make use of the archived data for fetching events.

Re: Elasticsearch and HDFS function in Metron (Hortonworks Cybersecurity Platform)

New Contributor

@asubramanian

i am wondering whats the demerit of just using Elasticsearch for short term and long term storage. will there be any problem in the future?

Re: Elasticsearch and HDFS function in Metron (Hortonworks Cybersecurity Platform)

Super Collaborator

@Rudy Hartono, I cannot think of an immediate demerit at this point. Metron is actively having a lot of features added, at a quick pace in that. If any feature comes up that makes use of the HDFS store, then there is a chance you will miss the functionality if you have disabled writing in the first place.

If you think my response helped, would you mind marking the answer as accepted? Thank you.