Member since
09-05-2016
21
Posts
11
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
743 | 09-29-2016 11:50 AM |
12-29-2016
03:54 PM
Hi Matt, yes i already hacked it with something similar. And now i am watching that issue. The logic in that piece of code is strange.
... View more
12-27-2016
12:38 PM
After i upgraded Nifi for some reason SplitText processor is failing with this error: Destination cannot be within sources.
I checked source code and i can see that it is caused by sources.contains(destination) in StandardProcessSession.java Anyone have any clue what could be the problem? Update: I just checked new install of nifi 1.1.1 and still i get this error. Scenario that i tested is next: GetFile -> SplitText -> LogAttribute. Update: Here is the issue related to this: https://issues.apache.org/jira/browse/NIFI-3255?jql=text%20~%20%22splittext%22
... View more
Labels:
11-01-2016
04:52 PM
I tried new installation of nifi, and just copied flowfile gz in conf. Still does not work, is the state also in components? Or just in the state state folder?
... View more
11-01-2016
04:17 PM
Hi Andrew, i tried to delete content of those directories. But still did not work. Even state of ListS3 was empty msg in debug was there is no new files yielding. I will try to delete all those directories completely, and see what will happen.
... View more
11-01-2016
03:44 PM
1 Kudo
Is there a way to clear a state of nifi? For example if i used ListS3 or Email processor. And i did some stuff, and they now now which files do they checked. Can i make them "forget"? Also is there a way to clean up/hard restart of nifi? To forget everything except flow file about processors and controllers. Best Bojan
... View more
Labels:
10-30-2016
08:25 PM
Hi Brad, Is this guide still valid? I can't get it to work. Best Bojan
... View more
10-19-2016
04:27 PM
I'm already using CSV -> AVRO -> JSON. Avro is better but problem is when you want to write to SQL. Also, with JSON you have JOLT, which can be useful for basic transformations. I'm not testing nifi with large amount of data, just some basic ETL stuff.
... View more
10-19-2016
03:34 PM
1 Kudo
Thank you all for responses, i will check the links and i hope after that will continue working on processor, and if i finish it i will publish it. CSV to JSON would be so nice to have. Best Bojan
... View more
10-17-2016
09:58 AM
1 Kudo
I'm developing custom processor, and i'm trying to use kitesdk in it. I added kite-data-core to my maven file but still i get errors like: Caused by: java.lang.ClassNotFoundException: org.apache.commons.lang.StringEscapeUtils After i add this dependency to my processor i get error for Logger dependency. And i did some searching about kite-sdk but i did not found anything. Is this a normal thing? Do i need all these dependencies for kitesdk. I'm building CSV to JSON processor, and the class that i use when i get these errors is CSVProperties. This is more a dev question. Best Bojan
... View more
Labels:
10-14-2016
09:22 AM
Thanks all for their opinions, and in the end i will do some work with ExecuteScript controller and JSON libs for Groovy. I wanted to use JOLT but unfortunately it can't use expression language. My idea was to change one field in flat JSON with some math operation like multiplying. And the multiplier should be from some other source. As for the custom processor that is always solution and it is not hard. But for this i will try ExecuteScript. Best Bojan
... View more
10-13-2016
09:51 AM
1 Kudo
I searched for some examples for basic mathematical transformations on FlowFile content( JSON columns for example), but i did not found any. I want to do some basic multiplication, addition etc. As i see it i can write custom JOLT transformer for this, or make it in SQL with stage tables or some other SQL way. Or to write new processor for this. But i was wondering what is the NIFI way to do this? Best Bojan
... View more
Labels:
10-11-2016
01:47 PM
Hi Bryan, This solved the problem, thank you. I was reading this https://nifi.apache.org/developer-guide.html, and i will check wiki also. Best Bojan
... View more
10-11-2016
01:25 PM
Hi, I am developing custom SQL nifi processor which will execute batch of SQL statements. It is not state of the art, but i just want to start developing. Problem is that my processor can't see active DBCPConnectionPool services. All other SQL components which are preinstalled in nifi can see it. I only have option to create new service, and when i do it complains that is not valid controller service, and i can see ID in property field. I'm clearly doing something wrong, but i cant find out what. Any hint is more then welcome.
... View more
Labels:
09-30-2016
02:33 PM
POP3 is also failing. In the end i created python script using smptlib to get messages after consumeimap fires. ConsumeImap -> ExecuteScript -> ExtractEmailAttachments But i don't like this solution, two time i am download messages.
... View more
09-30-2016
02:31 PM
Thanks for the patch. 🙂 I just tried to compile it and build is failing: validateConsumeIMAP
validateConsumeIMAPWithTimeout Chears! EDIT: I skipped the tests and just compiled. ConsumeImap processor now works as expected. Chears again!
... View more
09-29-2016
12:49 PM
No problem, i just need to create some hack. And good thing i know now how Email processors are working in NIFI(i read all the code). Maybe i can still use ConsumeIMAP processor to watch for new messages and then maybe route to ExecuteScript which will run Python code which will extract attachment and pass to next processor? Only issue that i see is the how to detect new messages, i am not sure if session is persistent or something like that. I could mark them as read after i download them. I did not have much experience with emails. But i will try. If you have suggestion how to hack this please be free to suggest.
... View more
09-29-2016
11:50 AM
3 Kudos
I think i hit this bug: https://issues.apache.org/jira/browse/NIFI-2709?jql=text%20~%20%22ConsumeImap%22
... View more
09-28-2016
03:28 PM
2 Kudos
Hi Brian, I get this from provenance:
--Apple-Mail=_851C8E02-DA29-4CEB-8309-895E2E5B1FB3
Content-Transfer-Encoding: 7bit
Content-Type: text/plain;
charset=us-ascii
Test body
--Apple-Mail=_851C8E02-DA29-4CEB-8309-895E2E5B1FB3
Content-Disposition: attachment;
filename=test.csv
Content-Type: text/csv;
name="test.csv"
Content-Transfer-Encoding: quoted-printable
foo;bar;;;;;=0D
1;2;;;;;=0D
3;4;;;;;=0D
5;6;;;;;=0D
;;;;;;=0D
;;;;;;=0D
;;;;;;=0D
;;;;;;=0D
;;;;;;=0D
;;;;;;=0D
;;;;;;=0D
;;;;;;=0D
;;;;;;=0D
;;;;;;=0D
;;;;;;=0D
;;;;;;=0D
;;;;;;=0D
;;;;;;=0D
;;;;;;=0D
;;;;;;=0D
;;;;;;=0D
;;;;;;=
--Apple-Mail=_851C8E02-DA29-4CEB-8309-895E2E5B1FB3--
... View more
09-28-2016
12:27 PM
2 Kudos
I'm trying to use ConsumeImap -> ExtractEmailAttachments processors with google mail. And it gets messages but fails with:
"Message failed RFC2822 validation". From source code i see there should be from and sentDate headers set. I was thinking something is wrong with gmail server and i wrote python code using imaplib and email and it works. This code is outside of nifi. Does anyone have any experience with this problem or clue how to fix this. Best Bojan
... View more
Labels:
09-06-2016
10:32 AM
Hallo, How do you handle restarts in Nifi? Is it possible to add new Processors or to swap jars without restarts? And if we need to restart how to handle them if we need high availability? It is possible to handle this on client side, but it is not solution. Best Bojan
... View more
Labels: