metron script prune_elasticsearch_indeces.sh

[#1]

Hi,

I have Metron cluster setup and it's been running. The data in ElasticSearch is starting to fill up pretty quickly and I wanted to use the functionality that came with Metron to prune ElasticSearch index. However, I could not find documentation regarding prune_elasticsearch_indeces.sh and i'm getting exception of ClassNotFound. If you have used this tool before or know how to run it properly or if I'm missing any installation for the missing library, please shed some lights. Thank you!

/usr/metron/0.4.0/bin/prune_elasticsearch_indices.sh -s 09/28/17 -n 2 -p bro_ -z master:2181,node-1:2181,node-2:2181

17/10/06 14:55:03 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-master:2181,hadoop-slave-1:2181,hadoop-slave-2:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@6a801d4b
17/10/06 14:55:03 INFO zookeeper.ClientCnxn: Opening socket connection to server hadoop-master/172.16.8.236:2181. Will not attempt to authenticate using SASL (unknown error)
17/10/06 14:55:03 INFO zookeeper.ClientCnxn: Socket connection established to hadoop-master/172.16.8.236:2181, initiating session
17/10/06 14:55:03 INFO zookeeper.ClientCnxn: Session establishment complete on server hadoop-master/172.16.8.236:2181, sessionid = 0x15ef200cb570461, negotiated timeout = 60000
17/10/06 14:55:03 INFO state.ConnectionStateManager: State change: CONNECTED
17/10/06 14:55:04 INFO elasticsearch.plugins: [Bloodstorm] modules [], plugins [], sites []
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.metron.guava.dataload.util.concurrent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor;
	at org.elasticsearch.threadpool.ThreadPool.<clinit>(ThreadPool.java:190)
	at org.elasticsearch.client.transport.TransportClient$Builder.build(TransportClient.java:131)
	at org.apache.metron.dataloads.bulk.ElasticsearchDataPrunerRunner.main(ElasticsearchDataPrunerRunner.java:103)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:148)

[#2 · Accepted solution]

Pruner will be deprecated in the next version of Metron. I would recommend using the Elastic Search curator to accomplish this.

https://github.com/elastic/curator

[#3]

@jsirota

Thank you for your reply and thank you for the information on the next release as well as available tool.

I found a workaround to delete in indexes through ElasticSearch plugin via json delete query.

Thank you!

[#4]

No problem. Please reach out if there is anything else we can help you with