Metron Threat intel flatfile_loader.sh gives me Table 'threatintel' was not found

deep_doradla — Tue, 20 Jun 2017 12:40:59 GMT

I am running a Metron cluster (v. 0.4.0.1.1.0.0) installed via Ambari.

While trying to load the threat intel feeds into Hbase using flatfile_loader.sh I am seeing the following error

This is what I ran

/usr/hcp/current/metron/bin/flatfile_loader.sh -n enrichment_config.json -i block_list.csv -t threatintel -c t -e extractor_config.json

Error:

Processing block_list.csv
Processed 3 - \17/06/20 05:59:37 ERROR client.AsyncProcess: Failed to get region location
org.apache.hadoop.hbase.TableNotFoundException: Table 'threatintel' was not found, got: profiler.
	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1284)
	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1165)
	at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:380)
	at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:331)
	at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:206)
	at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:183)
	at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1422)
	at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1025)
	at org.apache.metron.dataloads.nonbulk.flatfile.importer.LocalImporter.lambda$null$2(LocalImporter.java:111)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
	at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
	at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291)
	at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
17/06/20 05:59:37 ERROR client.AsyncProcess: Failed to get region location
org.apache.hadoop.hbase.TableNotFoundException: Table 'threatintel' was not found, got: profiler.
	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1284)
	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1165)
	at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:380)
	at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:331)
	at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:206)
	at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:183)
	at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1422)
	at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1025)
	at org.apache.metron.dataloads.nonbulk.flatfile.importer.LocalImporter.lambda$null$2(LocalImporter.java:111)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
	at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
	at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291)
	at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
17/06/20 05:59:37 ERROR client.AsyncProcess: Failed to get region location
org.apache.hadoop.hbase.TableNotFoundException: Table 'threatintel' was not found, got: profiler.
	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1284)
	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1165)
	at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:380)
	at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:331)
	at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:206)
	at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:183)
	at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1422)
	at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1025)
	at org.apache.metron.dataloads.nonbulk.flatfile.importer.LocalImporter.lambda$null$2(LocalImporter.java:111)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
	at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
	at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291)
	at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
	at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401)
	at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734)
	at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:160)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:174)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
	at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:583)
	at org.apache.metron.dataloads.nonbulk.flatfile.importer.LocalImporter.lambda$null$3(LocalImporter.java:108)
	at java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1386)
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
17/06/20 05:59:37 ERROR client.AsyncProcess: Failed to get region location
org.apache.hadoop.hbase.TableNotFoundException: Table 'threatintel' was not found, got: profiler.
	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1284)
	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1165)
	at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:380)
	at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:331)
	at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:206)
	at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:183)
	at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1422)
	at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1025)
	at org.apache.metron.dataloads.nonbulk.flatfile.importer.LocalImporter.lambda$null$2(LocalImporter.java:111)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
	at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
	at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291)
	at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)

From the above does it mean, a table name "threatintel" was supposed to be created when installing Metron but seems like this is missing in my case. If I have to create the table manually what is the schema for "threatintel"?

Any help is greatly appreciated.

Thanks

Deep

Re: Metron Threat intel flatfile_loader.sh gives me Table 'threatintel' was not found

deep_doradla — Tue, 20 Jun 2017 19:24:20 GMT

Just got this working. As the issue says, table not found. In these cases just create a table in hbase

create 'threatintel','t'

question Metron Threat intel flatfile_loader.sh gives me Table 'threatintel' was not found in Archives of Support Questions (Read Only)

Metron Threat intel flatfile_loader.sh gives me Table 'threatintel' was not found

Re: Metron Threat intel flatfile_loader.sh gives me Table 'threatintel' was not found