Support Questions
Find answers, ask questions, and share your expertise

Metron hdfs indexing bolt not working - No such file or directory error

Metron hdfs indexing bolt not working - No such file or directory error

Contributor

I am trying to run metron on my Ambari managed hcp cluster with hdp 2.5 & Metron 0.4.0. My parser works fine and I could see my squid messages coming to 'indexing' kafka topic. I have configured ES & Hdfs indexing. ES indexing works fine and am able to see alerts in metron UI. But when I checked the configured Hdfs directory there is nothing indexed. When I checked for storm hdfsindexing bolt it shows the below error

java.io.FileNotFoundException: 
/apps/metron/indexing/indexed/test/enrichment-hdfsIndexingBolt-3-0-1518011928845.json
 (No such file or directory)
	at java.io.FileOutputStream.open0(Native Method)
	at java.io.FileOutputStream.open(FileOutputStream.java:270)
	at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
	at java.io.FileOutputStream.<init>(FileOutputStream.java:101)
	at 
org.apache.metron.writer.hdfs.SourceHandler.createOutputFile(SourceHandler.java:156)
	at 
org.apache.metron.writer.hdfs.SourceHandler.initialize(SourceHandler.java:113)
	at 
org.apache.metron.writer.hdfs.SourceHandler.<init>(SourceHandler.java:68)
	at 
org.apache.metron.writer.hdfs.HdfsWriter.getSourceHandler(HdfsWriter.java:162)
	at org.apache.metron.writer.hdfs.HdfsWriter.write(HdfsWriter.java:112)
	at 
org.apache.metron.writer.BulkWriterComponent.flush(BulkWriterComponent.java:239)
	at 
org.apache.metron.writer.BulkWriterComponent.write(BulkWriterComponent.java:181)
	at 
org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:236)
	at 
org.apache.storm.daemon.executor$fn__6573$tuple_action_fn__6575.invoke(executor.clj:734)
	at 
org.apache.storm.daemon.executor$mk_task_receiver$fn__6494.invoke(executor.clj:466)
	at 
org.apache.storm.disruptor$clojure_handler$reify__6007.onEvent(disruptor.clj:40)
	at 
org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
	at 
org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
	at 
org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
	at 
org.apache.storm.daemon.executor$fn__6573$fn__6586$fn__6639.invoke(executor.clj:853)
	at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
	at clojure.lang.AFn.run(AFn.java:22)
	at java.lang.Thread.run(Thread.java:745) 

Will this issue be permission related ? I found this link. But I have verified that both storm & metron user can add files in the hdfs location /apps/metron/indexing/indexed/test/ ( test is my sensor configuration name ) when tried via command line. Please help.