rendi_7936
Explorer

How to use metadata in Apache Metron

by Explorer in Support Questions
‎06-21-2019 06:37 AM
‎06-21-2019 06:37 AM
Hello, Hi there, I need help about how to use metadata in Apache Metron. 1. Can you provide me a guidance how to use it ? 2. Is metadata have correlation with Kafka Key in Apache Nifi Processor (PutKafka and PublishKafka) ? I want to use metadata in Apache Metron because i want to implement multi-tenant as MSSP Context. I got this information from this question https://community.hortonworks.com/questions/185121/basic-questions-about-apache-metron.html . ... View more
Labels:

BLOOM_EXISTS(OBJECT_GET()) Cannot work in SENSOR E...

by Explorer in Support Questions
‎06-19-2019 05:35 PM
‎06-19-2019 05:35 PM
Hello, Hi there, I'm following this tutorial ( https://metron.apache.org/current-book/use-cases/typosquat_detection/index.html ). And in section Generate the Bloom Filter, i am very sure this command is working and return value true in Stellar command BLOOM_EXISTS(OBJECT_GET('/tmp/reference/alexa10k_filter.ser'), 'gogle') But, it does not work dan return an error when i put it in a Sensor at Apache Metron's Management -> RAW JSON -> SENSOR ENRICHMENT CONFIG. And i'm using ElasticSearch so i checked ELASTICSEARCH ENABLED. How i get the error, i use ElasticSearch HEAD as Google Chrome Addons ( https://chrome.google.com/webstore/detail/elasticsearch-head/ffmkiejjmecolpfloofpjologoblkegm ). And select error_index_ that have been created after Apache Metron send index to ElasticSearch. Software that i used Operating System - CentOS Linux release 7.6.1810 (Core) Hortonworks Software Used - Hortonworks Data Platform 2.6.5.0-292 - Hortonworks Data Flow 3.1.0.0 - Hortonworks CyberSecurity Platform 1.9.1.0 Software Used - Apache Kafka 1.0.0 - Apache Storm 1.1.0 - Apache Nifi 1.5.0 - Apache Metron 0.7.1.1.9.1.0 - HDFS 2.7.3 - HBase 1.1.2 Hardware Used : 4 (four) node using VMWare Virtualization, each using 4 core, 15.51 GB RAM, Storage 17.98 GB The Error : { "_index": "error_index_2019.06.19.10", "_type": "error_doc", "_id": "AWtt2DrFshKy3KgUuMHX", "_version": 1, "_score": 1, "_source": { "exception": "java.lang.IllegalStateException: ENRICHMENT error with stellar failed: Unable to parse: BLOOM_EXISTS(OBJECT_GET('/tmp/reference/alexa10k_filter.ser'), test_domain) due to: com.esotericsoftware.kryo.KryoException: Unable to find class: Serialization trace: bits (org.apache.metron.guava.enrichment.hash.BloomFilter) filter (org.apache.metron.stellar.common.utils.BloomFilter) with relevant variables test_domain=gogle", "failed_sensor_type": "error", "stack": "java.lang.IllegalStateException: ENRICHMENT error with stellar failed: Unable to parse: BLOOM_EXISTS(OBJECT_GET('/tmp/reference/alexa10k_filter.ser'), test_domain) due to: com.esotericsoftware.kryo.KryoException: Unable to find class: Serialization trace: bits (org.apache.metron.guava.enrichment.hash.BloomFilter) filter (org.apache.metron.stellar.common.utils.BloomFilter) with relevant variables test_domain=gogle at org.apache.metron.enrichment.parallel.ParallelEnricher.lambda$apply$0(ParallelEnricher.java:195) at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.metron.stellar.dsl.ParseException: Unable to parse: BLOOM_EXISTS(OBJECT_GET('/tmp/reference/alexa10k_filter.ser'), test_domain) due to: com.esotericsoftware.kryo.KryoException: Unable to find class: Serialization trace: bits (org.apache.metron.guava.enrichment.hash.BloomFilter) filter (org.apache.metron.stellar.common.utils.BloomFilter) with relevant variables test_domain=gogle at org.apache.metron.stellar.common.BaseStellarProcessor.createException(BaseStellarProcessor.java:173) at org.apache.metron.stellar.common.BaseStellarProcessor.parse(BaseStellarProcessor.java:154) at org.apache.metron.enrichment.adapters.stellar.StellarAdapter.process(StellarAdapter.java:118) at org.apache.metron.enrichment.adapters.stellar.StellarAdapter.enrich(StellarAdapter.java:190) at org.apache.metron.enrichment.adapters.stellar.StellarAdapter.enrich(StellarAdapter.java:43) at org.apache.metron.enrichment.parallel.EnrichmentCallable.apply(EnrichmentCallable.java:64) at org.apache.metron.enrichment.parallel.EnrichmentCallable.apply(EnrichmentCallable.java:30) at com.github.benmanes.caffeine.cache.LocalCache.lambda$statsAware$0(LocalCache.java:143) at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2039) at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853) at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2037) at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2020) at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:112) at com.github.benmanes.caffeine.cache.LocalManualCache.get(LocalManualCache.java:54) at org.apache.metron.enrichment.parallel.ParallelEnricher.lambda$apply$0(ParallelEnricher.java:182) ... 4 more Caused by: org.apache.metron.guava.enrichment.util.concurrent.UncheckedExecutionException: com.esotericsoftware.kryo.KryoException: Unable to find class: Serialization trace: bits (org.apache.metron.guava.enrichment.hash.BloomFilter) filter (org.apache.metron.stellar.common.utils.BloomFilter) at org.apache.metron.guava.enrichment.cache.LocalCache$Segment.get(LocalCache.java:2256) at org.apache.metron.guava.enrichment.cache.LocalCache.get(LocalCache.java:3980) at org.apache.metron.guava.enrichment.cache.LocalCache.getOrLoad(LocalCache.java:3984) at org.apache.metron.guava.enrichment.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4868) at org.apache.metron.enrichment.stellar.ObjectGet.apply(ObjectGet.java:110) at org.apache.metron.stellar.common.StellarCompiler.lambda$exitTransformationFunc$13(StellarCompiler.java:664) at org.apache.metron.stellar.common.StellarCompiler$Expression.apply(StellarCompiler.java:259) at org.apache.metron.stellar.common.BaseStellarProcessor.parse(BaseStellarProcessor.java:151) ... 17 more Caused by: com.esotericsoftware.kryo.KryoException: Unable to find class: Serialization trace: bits (org.apache.metron.guava.enrichment.hash.BloomFilter) filter (org.apache.metron.stellar.common.utils.BloomFilter) at com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:156) at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:133) at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:670) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:118) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551) at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:759) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:132) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790) at org.apache.metron.common.utils.SerDeUtils.fromBytes(SerDeUtils.java:262) at org.apache.metron.enrichment.stellar.ObjectGet$Loader.load(ObjectGet.java:87) at org.apache.metron.enrichment.stellar.ObjectGet$Loader.load(ObjectGet.java:72) at org.apache.metron.guava.enrichment.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3579) at org.apache.metron.guava.enrichment.cache.LocalCache$Segment.loadSync(LocalCache.java:2372) at org.apache.metron.guava.enrichment.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2335) at org.apache.metron.guava.enrichment.cache.LocalCache$Segment.get(LocalCache.java:2250) ... 24 more Caused by: java.lang.ClassNotFoundException: at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:154) ... 39 more ", "hostname": "HCP1-Master1", "source:type": "error", "raw_message": "{"": {"test_domain":null,"url":"www.gogle.com"} ,"source.type":"typo"}", "error_hash": "8219f37f7fd3f84b8dc6a2e04dd68b6139795b8e8fe4cc072ae9344ea559d1dc", "error_type": "enrichments_error", "guid": "6d295227-aa41-4a99-95e8-46dc5a938e1f", "message": "ENRICHMENT error with stellar failed: Unable to parse: BLOOM_EXISTS(OBJECT_GET('/tmp/reference/alexa10k_filter.ser'), test_domain) due to: com.esotericsoftware.kryo.KryoException: Unable to find class: Serialization trace: bits (org.apache.metron.guava.enrichment.hash.BloomFilter) filter (org.apache.metron.stellar.common.utils.BloomFilter) with relevant variables test_domain=gogle", "timestamp": 1560916000031} } Can you help me how to figure out what is going on. Thank in advance ... View more
Labels:

Performance Comparison Between Spark and Storm

by Explorer in Support Questions
‎09-06-2016 11:32 PM
‎09-06-2016 11:32 PM
Good Morning, guys Before i will go with the question, first i must to tell you this. I'm college student (IT) in Indonesia. This year is my final year and i want to make a project in Big Data. So, i want to get some data with Apache Nifi dan send it to Output. Afterwards, i process it with Spark and Storm. 1. What Performance Comparison should i do ? Response Time ? Throughput ? Real Time Proccessing Speed ? 2. Is it possible to do that ? Because, i know someone in the internet says benchmark between Spark and Storm inaccurate and there is no paper or journal about this. ( http://www.slideshare.net/ptgoetz/apache-storm-vs-spark-streaming ) 3. If you have a better idea about my final project. Let me know it. Thanks before, Have a nice day. ... View more

Fuzzy Algorithm in Apache Spark

by Explorer in Support Questions
‎08-29-2016 06:23 AM
‎08-29-2016 06:23 AM
Good Morning, everyone. Before i go with my question, i will start why i need fuzzy algorithm. In my project, i use Arduino to collect some data about temperature, etc. After i collect a lot of data, i want to use Fuzzy Algorithm to predict rainfall in some area. But, i dont find fuzzy algorithm in MLlib's Apache Spark ? What should i do ? How i can use fuzzy algorithm in HDF, or there is another way ? Or can i use python library in Apache Spark ? ; Because i heard there are some fuzzy library in Python, maybe i can use it in Apache Spark Thanks a lot. ... View more
Labels:

Re: Can i get twitter data to HDP, process it and ...

by Explorer in Support Questions
‎08-16-2016 06:44 AM
‎08-16-2016 06:44 AM
@Davide Isoardi Good Afternoon, sir. May i ask you something ? Is http://www.ecubecenter.it/Hadoop-Big-Data-Twitter-Map/ using this https://github.com/disoardi/tweetsdemo_with_ambari_views. It has been a while that i lost my mind to work my undergraduate project. Maybe, i will start again from this. Thanks before ... View more

Re: Machine Learning in Apache Nifi

by Explorer in Support Questions
‎05-02-2016 03:04 PM
‎05-02-2016 03:04 PM
@Ravi Mutyala Hmmm..., I dont know if there are more tutorial about Apache Nifi in this forum than Hortonworks Tutorial. Better to check it, and thanks a lot. ... View more

Re: Machine Learning in Apache Nifi

by Explorer in Support Questions
‎05-02-2016 03:01 PM
‎05-02-2016 03:01 PM
@Pierre Villard I want to send Twitter username from smartphone to HDP. When ListenHTTP get the data, it will send to GetTwitter to get some tweet according username. After that, i will use spark to calculate Machine Learning Algorithm to predict user personality. ... View more

Re: Machine Learning in Apache Nifi

by Explorer in Support Questions
‎05-02-2016 02:57 PM
‎05-02-2016 02:57 PM
@Kirk Haslbeck Thanks, i will try it, one of them as fast as possible ... View more

Machine Learning in Apache Nifi

by Explorer in Support Questions
‎04-30-2016 02:53 AM
3 Kudos
‎04-30-2016 02:53 AM
3 Kudos
Hello guys, First, i want to make a project with Apache Nifi, but there is some problem that i face it now. 1. Can i make Machine Learning Calculation in Apache Nifi ? I know that i must use Mahout and Spark, but i dont find Mahout and Spark Processor in Apache Nifi. What should i do ? Can you suggest me another idea. 2. I want to make web service with Apache Nifi ? First time, i use ListenHTTP Processor, but the text that i send become Flow File. How can i convert it to text so i can put it in GetTwitter Processor ? Thanks for reply 🙂 ... View more

Re: Can i get twitter data to HDP, process it and ...

by Explorer in Support Questions
‎04-21-2016 09:20 AM
‎04-21-2016 09:20 AM
Hmmmm, so the key of this is Apache Nifi. In Nifi, we can create a sequence from get the data, process it and save it according the Processor what we choose. If in the first time, i can load "aloha.csv" and make a table from it manually (Click Files Views and Hive View). So, with Nifi i can load "aloha.csv" and make a table automatically using the Processor. Is it correct ? Well, maybe i try it. I'm sorry if i ask too much because i need to clarify this. Because, if i cant implement this (my final project in the college), i will step back to search another idea. ... View more

Re: Can i get twitter data to HDP, process it and ...

by Explorer in Support Questions
‎04-21-2016 02:32 AM
‎04-21-2016 02:32 AM
@Davide Isoardi Thanks for reply my question. But, i have another question. When an App using web service, the sequence will be like this. Android --[JSON Data]--> Web Service --[SQL Query]--> Database. In this case, to proccess the data, we only using Web Service and send SQL Query to Database. From the your answer this is what i get. Android --[Twitter Data]--> Nifi --> Solr --> Spark --> Another Solr Collection --[JSON Format]--> Android In this case, to process the data we need many app to send data. From Android to Nifi, from Nifi to Solr, etc. Can Solr receive data from Nifi after that send to Spark ? Because i never know how to communicate (send/receive data) from one app to another app in HDP automatically like your idea ? I'm sorry, if my question makes you confuse 😉 ... View more

Re: Can i get twitter data to HDP, process it and ...

by Explorer in Support Questions
‎04-21-2016 01:37 AM
‎04-21-2016 01:37 AM
@Andrea D'Orio Thanks for your support 😉 ... View more

Can i get twitter data to HDP, process it and show...

by Explorer in Support Questions
‎04-20-2016 08:43 AM
1 Kudo
‎04-20-2016 08:43 AM
1 Kudo
Hello anyone, First, i want say sorry because maybe my question is out of the topic. But, i need to ask it because i dont find another forum that discuss about Big Data and how to use it with your own (Using Tutorial that provided by Hortonworks). My question is 1. Can i get data from twitter to HDP ? Is it true that i can use Apache Nifi or Can i use another Software like Apache Kafka ? But How ? 2. When i get the data form Twitter, how to process it ? Can i use Spark or Mahout ? In this case, i want to find the most people personality from twitter, which is the best option ? 3. Then, if i can calculate the data, how i can access it with android aplication ? In my mind, i think i can use web service but i dont know how to make it with HDP ? I will make a clear explanation about this, a. User want to use my app. But, user must login to Twitter before use this app. b. After this, i want to collect some twitter data from user. c. Twitter data passes to HDP using Web services. d. HDP will calculate the data with certain algorithm. e. After that, HDP will send the result the data to android app. f. Voila, use can see his/her personality using twitter data. Can i implement this ? Or maybe i think too much how use HDP ? 4. Do you some another idea for my final project in the college ? Actually, my older lecturer use HDP but only single cluster (Sandbox). And i want to make it multi node, but i dont know how to maximize the potential how to use HDP and real Bid Data. I hope my question with not confuse you guys, why i ask to many question because my lack knowledege how to use HDP and Big Data. I hope i can get the answer and have a nice day :). ... View more
Contact Me
Online
Offline
Last Visited
‎07-01-2019 03:03 AM
Public Statistics
Date Registered ‎04-19-2016 10:57 PM
Last Visited ‎07-01-2019 03:03 AM
Total Messages Posted 13
Total Kudos Received 4