About blood9raven

blood9raven · ‎12-29-2016

Hi Matt, yes i already hacked it with something similar. And now i am watching that issue. The logic in that piece of code is strange.

blood9raven · ‎12-27-2016

After i upgraded Nifi for some reason SplitText processor is failing with this error: Destination cannot be within sources. I checked source code and i can see that it is caused by sources.contains(destination) in StandardProcessSession.java Anyone have any clue what could be the problem? Update: I just checked new install of nifi 1.1.1 and still i get this error. Scenario that i tested is next: GetFile -> SplitText -> LogAttribute. Update: Here is the issue related to this: https://issues.apache.org/jira/browse/NIFI-3255?jql=text%20~%20%22splittext%22

blood9raven · ‎11-01-2016

I tried new installation of nifi, and just copied flowfile gz in conf. Still does not work, is the state also in components? Or just in the state state folder?

blood9raven · ‎11-01-2016

Hi Andrew, i tried to delete content of those directories. But still did not work. Even state of ListS3 was empty msg in debug was there is no new files yielding. I will try to delete all those directories completely, and see what will happen.

blood9raven · ‎11-01-2016

Is there a way to clear a state of nifi? For example if i used ListS3 or Email processor. And i did some stuff, and they now now which files do they checked. Can i make them "forget"? Also is there a way to clean up/hard restart of nifi? To forget everything except flow file about processors and controllers. Best Bojan

blood9raven · ‎10-30-2016

Hi Brad, Is this guide still valid? I can't get it to work. Best Bojan

blood9raven · ‎10-19-2016

I'm already using CSV -> AVRO -> JSON. Avro is better but problem is when you want to write to SQL. Also, with JSON you have JOLT, which can be useful for basic transformations. I'm not testing nifi with large amount of data, just some basic ETL stuff.

blood9raven · ‎10-19-2016

Thank you all for responses, i will check the links and i hope after that will continue working on processor, and if i finish it i will publish it. CSV to JSON would be so nice to have. Best Bojan

blood9raven · ‎10-17-2016

I'm developing custom processor, and i'm trying to use kitesdk in it. I added kite-data-core to my maven file but still i get errors like: Caused by: java.lang.ClassNotFoundException: org.apache.commons.lang.StringEscapeUtils After i add this dependency to my processor i get error for Logger dependency. And i did some searching about kite-sdk but i did not found anything. Is this a normal thing? Do i need all these dependencies for kitesdk. I'm building CSV to JSON processor, and the class that i use when i get these errors is CSVProperties. This is more a dev question. Best Bojan

blood9raven · ‎10-14-2016

Thanks all for their opinions, and in the end i will do some work with ExecuteScript controller and JSON libs for Groovy. I wanted to use JOLT but unfortunately it can't use expression language. My idea was to change one field in flat JSON with some math operation like multiplying. And the multiplier should be from some other source. As for the custom processor that is always solution and it is not hard. But for this i will try ExecuteScript. Best Bojan

blood9raven · ‎10-13-2016

I searched for some examples for basic mathematical transformations on FlowFile content( JSON columns for example), but i did not found any. I want to do some basic multiplication, addition etc. As i see it i can write custom JOLT transformer for this, or make it in SQL with stage tables or some other SQL way. Or to write new processor for this. But i was wondering what is the NIFI way to do this? Best Bojan

blood9raven · ‎10-11-2016

Hi Bryan, This solved the problem, thank you. I was reading this https://nifi.apache.org/developer-guide.html, and i will check wiki also. Best Bojan

blood9raven · ‎10-11-2016

Hi, I am developing custom SQL nifi processor which will execute batch of SQL statements. It is not state of the art, but i just want to start developing. Problem is that my processor can't see active DBCPConnectionPool services. All other SQL components which are preinstalled in nifi can see it. I only have option to create new service, and when i do it complains that is not valid controller service, and i can see ID in property field. I'm clearly doing something wrong, but i cant find out what. Any hint is more then welcome.

blood9raven · ‎09-30-2016

POP3 is also failing. In the end i created python script using smptlib to get messages after consumeimap fires. ConsumeImap -> ExecuteScript -> ExtractEmailAttachments But i don't like this solution, two time i am download messages.

blood9raven · ‎09-30-2016

Thanks for the patch. 🙂 I just tried to compile it and build is failing: validateConsumeIMAP validateConsumeIMAPWithTimeout Chears! EDIT: I skipped the tests and just compiled. ConsumeImap processor now works as expected. Chears again!

blood9raven · ‎09-29-2016

No problem, i just need to create some hack. And good thing i know now how Email processors are working in NIFI(i read all the code). Maybe i can still use ConsumeIMAP processor to watch for new messages and then maybe route to ExecuteScript which will run Python code which will extract attachment and pass to next processor? Only issue that i see is the how to detect new messages, i am not sure if session is persistent or something like that. I could mark them as read after i download them. I did not have much experience with emails. But i will try. If you have suggestion how to hack this please be free to suggest.

blood9raven · ‎09-29-2016

I think i hit this bug: https://issues.apache.org/jira/browse/NIFI-2709?jql=text%20~%20%22ConsumeImap%22

blood9raven · ‎09-28-2016

Hi Brian, I get this from provenance: --Apple-Mail=_851C8E02-DA29-4CEB-8309-895E2E5B1FB3 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii Test body --Apple-Mail=_851C8E02-DA29-4CEB-8309-895E2E5B1FB3 Content-Disposition: attachment; filename=test.csv Content-Type: text/csv; name="test.csv" Content-Transfer-Encoding: quoted-printable foo;bar;;;;;=0D 1;2;;;;;=0D 3;4;;;;;=0D 5;6;;;;;=0D ;;;;;;=0D ;;;;;;=0D ;;;;;;=0D ;;;;;;=0D ;;;;;;=0D ;;;;;;=0D ;;;;;;=0D ;;;;;;=0D ;;;;;;=0D ;;;;;;=0D ;;;;;;=0D ;;;;;;=0D ;;;;;;=0D ;;;;;;=0D ;;;;;;=0D ;;;;;;=0D ;;;;;;=0D ;;;;;;= --Apple-Mail=_851C8E02-DA29-4CEB-8309-895E2E5B1FB3--

blood9raven · ‎09-28-2016

I'm trying to use ConsumeImap -> ExtractEmailAttachments processors with google mail. And it gets messages but fails with: "Message failed RFC2822 validation". From source code i see there should be from and sentDate headers set. I was thinking something is wrong with gmail server and i wrote python code using imaplib and email and it works. This code is outside of nifi. Does anyone have any experience with this problem or clue how to fix this. Best Bojan

blood9raven · ‎09-09-2016

Thanks, i will check out clustering. Best Bojan

blood9raven · ‎09-06-2016

Hallo, How do you handle restarts in Nifi? Is it possible to add new Processors or to swap jars without restarts? And if we need to restart how to handle them if we need high availability? It is possible to handle this on client side, but it is not solution. Best Bojan

Online	Offline
Last Visited	‎07-22-2017 02:22 PM

Date Registered	‎09-05-2016 03:36 PM
Last Visited	‎07-22-2017 02:22 PM
Total Messages Posted	21
Total Kudos Received	11

Re: RFC 2822 Validation error

Re: Issues with nifi upgrade:Destination cannot be...

Issues with nifi upgrade:Destination cannot be wit...

Re: Nifi clear state

Re: Nifi clear state

Nifi clear state

Re: Securing NiFi Step-by-Step

Re: KiteSDK integration

Re: KiteSDK integration

KiteSDK integration

Re: Mathematical operations on flowfile

Mathematical operations on flowfile

Re: Custom Nifi Processor can't see existing Contr...

Custom Nifi Processor can't see existing Controlle...

Re: RFC 2822 Validation error

Re: RFC 2822 Validation error

Re: RFC 2822 Validation error

Re: RFC 2822 Validation error

Re: RFC 2822 Validation error

RFC 2822 Validation error

Re: Hot swapping the jar in Nifi

Hot swapping the jar in Nifi