Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

metron script prune_elasticsearch_indeces.sh

avatar
Expert Contributor

Hi,

I have Metron cluster setup and it's been running. The data in ElasticSearch is starting to fill up pretty quickly and I wanted to use the functionality that came with Metron to prune ElasticSearch index. However, I could not find documentation regarding prune_elasticsearch_indeces.sh and i'm getting exception of ClassNotFound. If you have used this tool before or know how to run it properly or if I'm missing any installation for the missing library, please shed some lights. Thank you!

/usr/metron/0.4.0/bin/prune_elasticsearch_indices.sh -s 09/28/17 -n 2 -p bro_ -z master:2181,node-1:2181,node-2:2181
17/10/06 14:55:03 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-master:2181,hadoop-slave-1:2181,hadoop-slave-2:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@6a801d4b
17/10/06 14:55:03 INFO zookeeper.ClientCnxn: Opening socket connection to server hadoop-master/172.16.8.236:2181. Will not attempt to authenticate using SASL (unknown error)
17/10/06 14:55:03 INFO zookeeper.ClientCnxn: Socket connection established to hadoop-master/172.16.8.236:2181, initiating session
17/10/06 14:55:03 INFO zookeeper.ClientCnxn: Session establishment complete on server hadoop-master/172.16.8.236:2181, sessionid = 0x15ef200cb570461, negotiated timeout = 60000
17/10/06 14:55:03 INFO state.ConnectionStateManager: State change: CONNECTED
17/10/06 14:55:04 INFO elasticsearch.plugins: [Bloodstorm] modules [], plugins [], sites []
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.metron.guava.dataload.util.concurrent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor;
	at org.elasticsearch.threadpool.ThreadPool.<clinit>(ThreadPool.java:190)
	at org.elasticsearch.client.transport.TransportClient$Builder.build(TransportClient.java:131)
	at org.apache.metron.dataloads.bulk.ElasticsearchDataPrunerRunner.main(ElasticsearchDataPrunerRunner.java:103)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:148)

1 ACCEPTED SOLUTION

avatar
Contributor

Pruner will be deprecated in the next version of Metron. I would recommend using the Elastic Search curator to accomplish this.

https://github.com/elastic/curator

View solution in original post

3 REPLIES 3

avatar
Contributor

Pruner will be deprecated in the next version of Metron. I would recommend using the Elastic Search curator to accomplish this.

https://github.com/elastic/curator

avatar
Expert Contributor

@jsirota

Thank you for your reply and thank you for the information on the next release as well as available tool.

I found a workaround to delete in indexes through ElasticSearch plugin via json delete query.

Thank you!

avatar
Contributor

No problem. Please reach out if there is anything else we can help you with