Member since
06-04-2016
4
Posts
0
Kudos Received
0
Solutions
06-11-2016
03:37 PM
@Paul Boal, I found it as easy as you say in maven on the command line. And actually, I'm at the point whee the code is ready to be moved from testing into a custom processor-but... I cannot get PutKafka to work. Without running any of my code, just a: With a known good Kafka running (other /producers using the same Topic)... ...yields: Auto refresh started
11:17:19 EDT
WARNING
2b76634c-6354-4c3c-b977-86f73d64efbc
PutKafka[id=2b76634c-6354-4c3c-b977-86f73d64efbc] Processor Administratively Yielded for 1 sec due to processing failure
11:18:00 EDT
ERROR
2b76634c-6354-4c3c-b977-86f73d64efbc
PutKafka[id=2b76634c-6354-4c3c-b977-86f73d64efbc] Failed while waiting for acks from Kafka
11:18:00 EDT
ERROR
2b76634c-6354-4c3c-b977-86f73d64efbc
PutKafka[id=2b76634c-6354-4c3c-b977-86f73d64efbc] PutKafka[id=2b76634c-6354-4c3c-b977-86f73d64efbc] failed to process due to java.lang.NullPointerException; rolling back session: java.lang.NullPointerException
11:18:00 EDT
ERROR
2b76634c-6354-4c3c-b977-86f73d64efbc
PutKafka[id=2b76634c-6354-4c3c-b977-86f73d64efbc] PutKafka[id=2b76634c-6354-4c3c-b977-86f73d64efbc] failed to process session due to java.lang.NullPointerException: java.lang.NullPointerException I'll post all this to another forum & thread, but I will say that when creating something new, and coming across a problem, my first [incorrect] instinct is to troubleshoot my own code... In this case, it was not my code! 🙂
... View more
06-08-2016
02:34 AM
Paul, I did! And although it was not possible in stock intellij (I couldn't figure out which resource to add to the archetype repo), it DOES work with maven on the command line, then load project in intellij... My current problem is finding time to do the same thing with ControllerServices... they may end up expensive, but would be threadsafe... REST or other web hosting may end up working, but this will take testing, as there is no way I want to load the Stanford or Google text parsers for every request... I'll keep everyone abreast of what I discover
... View more
06-05-2016
01:18 AM
@jfrazee I am very impressed and grateful for your response. To date I was thinking I'd have to use Kafka (as in Java Kafka Consumer), as Nifi is good with Kafka out of the box... I can envision 100s of events per second quite easily... Kafka's consumer groups might help with longer running parsers, effectively adding a pool of parsers who can then feed their results back into Nifi via Kafka again. Looking at the nar requirements for a nifi service and accompanying processor looks like a steep learning curve for me at this time... I owe it to you to look into each of these pointers carefully though, and over this coming week I will! Thanks again!
... View more
06-04-2016
09:47 PM
I realize the program could be a web, FTP, etc. server, but I'm talking about directly & efficiently. Having to start a program over & over again seems a huge waste when throughput goes up. Imagine calling a running Java program that is in a while() loop, looping through on each line from STDIN, using a newline as the delimiter. ExecuteStreamCommand seems to just call a program/script, optionally passing the flowfile on STDIN, but again, it is asking to load the program from disk & GC for every single flowfile... Can we send data to that program lightning fast without having to setup web, SFTP, SSH servers? --- Again, the use case is a processor being hammered with text as a flowfile, and wanting to get it into a java program as fast as possible. Loading, running, and stopping the code for each and every flowfile is just too computationally expensive. Preferred is a direct connection to the application, as when the application is running continuously, blocking when no data. Your insights appreciated! --- We're not talking about calling a static method, but this at first seemed interesting: from here: "Answer by mburgess Apr 27 at 03:34 AM To @Artem Ervits comment, if the Java class is available in the class loader (either by being part of the NiFi Core or being specified as a JAR in the ExecuteScript's Module Directory property), then you can call it directly, you don't need reflection. If I'm not answering your question, can you provide more details? I'm happy to help :)"
... View more
Labels:
- Labels:
-
Apache NiFi