metron script prune_elasticsearch_indeces.sh

nnat25191 — Sat, 07 Oct 2017 02:01:40 GMT

Hi,

I have Metron cluster setup and it's been running. The data in ElasticSearch is starting to fill up pretty quickly and I wanted to use the functionality that came with Metron to prune ElasticSearch index. However, I could not find documentation regarding prune_elasticsearch_indeces.sh and i'm getting exception of ClassNotFound. If you have used this tool before or know how to run it properly or if I'm missing any installation for the missing library, please shed some lights. Thank you!

/usr/metron/0.4.0/bin/prune_elasticsearch_indices.sh -s 09/28/17 -n 2 -p bro_ -z master:2181,node-1:2181,node-2:2181

17/10/06 14:55:03 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-master:2181,hadoop-slave-1:2181,hadoop-slave-2:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@6a801d4b
17/10/06 14:55:03 INFO zookeeper.ClientCnxn: Opening socket connection to server hadoop-master/172.16.8.236:2181. Will not attempt to authenticate using SASL (unknown error)
17/10/06 14:55:03 INFO zookeeper.ClientCnxn: Socket connection established to hadoop-master/172.16.8.236:2181, initiating session
17/10/06 14:55:03 INFO zookeeper.ClientCnxn: Session establishment complete on server hadoop-master/172.16.8.236:2181, sessionid = 0x15ef200cb570461, negotiated timeout = 60000
17/10/06 14:55:03 INFO state.ConnectionStateManager: State change: CONNECTED
17/10/06 14:55:04 INFO elasticsearch.plugins: [Bloodstorm] modules [], plugins [], sites []
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.metron.guava.dataload.util.concurrent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor;
	at org.elasticsearch.threadpool.ThreadPool.<clinit>(ThreadPool.java:190)
	at org.elasticsearch.client.transport.TransportClient$Builder.build(TransportClient.java:131)
	at org.apache.metron.dataloads.bulk.ElasticsearchDataPrunerRunner.main(ElasticsearchDataPrunerRunner.java:103)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:148)

Re: metron script prune_elasticsearch_indeces.sh

jsirota — Sat, 28 Oct 2017 02:24:41 GMT

Pruner will be deprecated in the next version of Metron. I would recommend using the Elastic Search curator to accomplish this.

https://github.com/elastic/curator

Re: metron script prune_elasticsearch_indeces.sh

nnat25191 — Thu, 02 Nov 2017 02:11:47 GMT

@jsirota

Thank you for your reply and thank you for the information on the next release as well as available tool.

I found a workaround to delete in indexes through ElasticSearch plugin via json delete query.

Thank you!

Re: metron script prune_elasticsearch_indeces.sh

jsirota — Sat, 04 Nov 2017 01:18:34 GMT

No problem. Please reach out if there is anything else we can help you with

question Re: metron script prune_elasticsearch_indeces.sh in Archives of Support Questions (Read Only)

metron script prune_elasticsearch_indeces.sh

Re: metron script prune_elasticsearch_indeces.sh

Re: metron script prune_elasticsearch_indeces.sh

Re: metron script prune_elasticsearch_indeces.sh