About jknulst

jknulst · ‎05-01-2018

Great article!

jknulst · ‎03-13-2018

@Matt Burgess I don't get the syntax, but it works like a charm! Thank you

jknulst · ‎03-13-2018

@Rahul Soni Thanks for your answer but I really don't want to hardcode all of my 50+ JSON fields

jknulst · ‎03-13-2018

@Shu Maybe, one more thing: -I would preferably don't want to do any hardcoding, since the JSON has many keys I am specifically looking for a dynamic way I came across JOLT functions like =toLower / =join and was kind of hoping for =replace but that one seems missing.

jknulst · ‎03-13-2018

Hi, I have a requirement for transforming JSON on Nifi which seems simple but I haven't been able to solve: Input json: { "agent-submit-time" : -1, "agent-end-time" : 123445, "agent-name" : "Marie Bayer-Smith" } Desired: { "agent_submit_time" : -1, "agent_end_time" : 123445, "agent_name" : "Marie Bayer-Smith" } -I don't want to use the ReplaceText processor since replacing "-" for "_" might impact values too -I need this to be able to infer AVRO schema on incoming JSON records (AVRO does not like the dashes at all) -since I already use a Jolt processor for another transformation in the JSON it makes sense to include it in the same processor to prevent unnecessary Nifi overhead. I think I would need the JoltTransformJSON processor as it is very powerful (but the syntax evades me) for this but open for other options too.

jknulst · ‎01-04-2018

Updated. Maybe you need a github account

jknulst · ‎01-04-2018

Please have a look at the documentation. For instance here : https://github.com/apache/metron/tree/master/metron-platform/metron-parsers and here: https://github.com/apache/metron/tree/master/metron-platform/metron-enrichment https://github.com/apache/metron/tree/master/metron-platform/metron-indexing Indexing to ES is supported in all Metron versions. If you want the latest and greatest go for HCP-1.4 / Metron-0.4.1 ( https://docs.hortonworks.com/HDPDocuments/HCP1/HCP-1.4.0/index.html )

jknulst · ‎01-03-2018

@Gaurav Bapat Your Nifi setup looks weird. Why do you have 2 PutKafka processors? Look for error messages on the PutKafka processor. Those will tell why the syslog event are not making it into the Kafka topic. (you created a target topic, right?) I think that at the time of writing that tutorial, the parsing topology was directly writing/indexing into ES. That is not the case anymore in newer versions of Metron. You will have to spin up the indexing topology as well for the same effect, and have indexing source from the Kafka topic that you have as a destination Kafka topic for the parsing topology. You can set that up via Ambari.

jknulst · ‎01-02-2018

@Gaurav Bapat On single node just spin up HDFS, Zookeeper, Kafka, Storm, ES and Kibana (and Metron). HBase is only necessary when you do lookups during the enrichment topology. Shut down all other services. Ambari will be helpful in settings the memory allocations based on what is available to the node. Sometimes you can set them even lower then the recommended settings. When the VM has less than 8 GB available you will have a real hard time to run Metron at all. Although it is fairly dated by now, this tutorial would still take you through the required steps to see events pop up in Kibana: https://cwiki.apache.org/confluence/display/METRON/2016/04/25/Metron+Tutorial+-+Fundamentals+Part+1%3A+Creating+a+New+Telemetry

jknulst · ‎01-02-2018

Hi Gaurav, When you are running all on a single node it is very common that multiple services are fighting for (scarce) available memory. Services like Hbase tend to fail when this happens. You could try to be selective on what services you run concurrently. For instance if you just want to run Metron services, shut down Ambari Metrics, Oozie, Spark, Zeppelin and maybe Yarn and Hive cause you don't need it. You have to compromise when running single node. For Metron make sure HDFS, Zookeeper, HBase, Kafka, Storm, ES and Kibana are up and OK.

Online	Offline
Last Visited	‎12-05-2024 10:10 AM

Member Since	‎08-15-2016 02:39 PM
Last Visited	‎12-05-2024 10:10 AM
Posts	189
Kudos received	63

Cloudera Community

Re: Metron Connection failed [Errno 111] error on ...

Re: How do I verify all of my data from enrichment...

Re: Metron profiler client unbale to load native h...

Re: Kafka ConsumerGroupCommand Error

Re: Metron Parser Fail to start with Ambari 2.5 an...

Re: Ambari Single Sign On with Okta and KnoxSSO, p...

Re: How to do JOLT replace on all JSON keys in Nif...

Re: How to do JOLT replace on all JSON keys in Nif...

Re: How to do JOLT replace on all JSON keys in Nif...

How to do JOLT replace on all JSON keys in Nifi

Re: Metron Connection failed [Errno 111] error on ...

Re: Metron Connection failed [Errno 111] error on ...

Re: Metron Connection failed [Errno 111] error on ...

Re: Metron Connection failed [Errno 111] error on ...

Re: Metron Connection failed [Errno 111] error on ...