Re: get flowfile for provenance event

alfonz19 · ‎01-19-2018

Alright. So now I need help with what-to-do-next, as I have no idea what is proper way how to do that in nifi. I have Json and Avro. I want to validate Json using Avro (ValidateRecord processor?) and if it fails, I want to route FlowFile along with description what went wrong to failure relationship. Currently there is this description emited as provenance file. I have modified code of ValidateRecord, I can use that if there isn't better way. I thought the provenance events is the way, a because of that I closed my pull request. What is your recommendation?

alfonz19 · ‎01-19-2018

Hi, I'm using siteToSiteProvenanceReportingTask. In my flow I can then read provenance event, which references some FlowFile. But how to access that flowFile? thanks, M.

alfonz19 · ‎01-18-2018

Hi, If my flowfile contains json, which at JsonPath does not have array, it's routed to failure. But lets assume, that I don't know, whether I get single record or array of recods. Would it make sense to create property to control behavior in such case? User could then decide, whether it's an error, or whether unchanged flow should be routed to original relationship (or any other relationship) Or is there any other way, how to test flowfile, whether there is at given JsonPath array or something else? Thanks! (if change described in first paragraph is necessary and would be accepted, I can do the coding, np).

alfonz19 · ‎01-16-2018

Ok, thanks for explanation. I did not know about penalization before, and that it is called during rollback. Thanks for info.

alfonz19 · ‎01-16-2018

I understand that situation about transaction. I even understand that there can theoretically be event, which yields failure, which is temporary. However based on what I saw in my life, I'd consider that as an exception. And even if lets say remote connection or other IO can be 'healed' after a while, so next retry can succeed, I pretty much doubt, that 6 retries in one millisecond(from my log file) has any realistic chance of succeeding. Based on lets say opinion, that everything can break and there can be 'hole' anywhere, I'd suggest several improvements maybe: a) if user confirms that order is not important, we can move flowfile a little bit to back giving it some time and chance for others. b) if that is not possible, and we have to live with poison flowfile, we can at least do some throttling and logarithmically slowing down consuming event. It's fine to do first 10 retries in few ms, but doing 1000th retry in same pace is not required I believe. After it was proven, that poison flowfile blocking processing won't pass, it could be even considered stopping processor. Just idea. About code style; I'm curious what your code style is, so I can adhere to it. For example some people believe, that making method static gives some performance benefit, so I'm not criticizing, I'm asking why it's like that. visibility modifier example: org.apache.nifi.processors.standard.EvaluateJsonPath#PATH_NOT_FOUND_IGNORE 'invalid' static org.apache.nifi.processors.standard.AbstractJsonPathProcessor#isJsonScalar

alfonz19 · ‎01-16-2018

Hi, I'd like to ask two questions. First one about error handling in nifi, which worries me a little: Lets say I have custom processor, which triggers on FlowFile arrival. But that processor is buggy, and throws exception before flow file is removed. Exception is logged, flowfile not removed from queue and it's presence will trigger buggy processor again. Indefinitely. Am I getting it right? Since it seems it works like this. If it is, can you explain why you opted for this? Since guaranteeing, that every processor is bug-free is high bet. But if it's not like that, what is the recommended error handling? --- second question is about style, I'm hijacking this topic a little, but it's similarly generic question. In code I see lots of static methods, which does not have to be static at all. Stuff which can be private, but it's public, or package-local. Why's that? thanks!

alfonz19 · ‎01-15-2018

Thanks a lot! This helped me, based on this hint everything works as it should. Thanks again. Martin.

alfonz19 · ‎01-12-2018

after immense trial&error rounds, we moved forward and currently face this. If we do not provide this dependency in pom.xml <dependency> <groupId>org.apache.nifi</groupId> <artifactId>nifi-record-serialization-service-api</artifactId> <scope>compile</scope> </dependency> then nar cannot be deployed, because those interfaces are missing. If we leave out compile, nar cannot be deployed, since interfaces are not on classpath. If we keep compile there, interface will be there, but it will be considered to be different version and implementators should extend our interface defined in out built nar. Is this a bug or what? I assume I should be able to add just: <dependency> <groupId>org.apache.nifi</groupId> <artifactId>nifi-record-serialization-service-api</artifactId> </dependency> or what am I doing wrong? EDIT: current pom.xms: • packaging pom does not have anything interresting • pom.xml of nar module <?xml version="1.0" encoding="UTF-8"?>  <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <parent> <groupId>???</groupId> <artifactId>???</artifactId> <version>1.0-SNAPSHOT</version> </parent> <artifactId>???</artifactId> <version>1.0-SNAPSHOT</version> <packaging>nar</packaging> <properties> <maven.javadoc.skip>true</maven.javadoc.skip> <source.skip>true</source.skip> </properties> <dependencies> <dependency> <groupId>org.apache.nifi</groupId> <artifactId>nifi-record</artifactId> <scope>compile</scope> </dependency> <dependency> <groupId>org.apache.nifi</groupId> <artifactId>nifi-record-serialization-service-api</artifactId> <scope>compile</scope> </dependency> <dependency> <groupId>org.apache.nifi</groupId> <artifactId>nifi-schema-registry-service-api</artifactId> <scope>compile</scope> </dependency> <dependency> <groupId>cz.embedit</groupId> <artifactId>nifi-fdp-processors</artifactId> <version>1.0-SNAPSHOT</version> </dependency> </dependencies> </project> • pom.xml of processors module <?xml version="1.0" encoding="UTF-8"?>  <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <parent> <groupId>???</groupId> <artifactId>???</artifactId> <version>1.0-SNAPSHOT</version> </parent> <artifactId>???</artifactId> <packaging>jar</packaging> <dependencies> <dependency> <groupId>org.apache.nifi</groupId> <artifactId>nifi-standard-processors</artifactId> <version>${nifi.version}</version> <scope>provided</scope> </dependency> <dependency> <groupId>org.apache.nifi</groupId> <artifactId>nifi-record-serialization-service-api</artifactId> <scope>compile</scope> </dependency> <dependency> <groupId>org.apache.nifi</groupId> <artifactId>nifi-avro-record-utils</artifactId> <scope>compile</scope> </dependency> <dependency> <groupId>org.apache.nifi</groupId> <artifactId>nifi-schema-registry-service-api</artifactId> <scope>compile</scope> </dependency> <dependency> <groupId>org.apache.nifi</groupId> <artifactId>nifi-record</artifactId> <scope>compile</scope> </dependency> <dependency> <groupId>org.apache.nifi</groupId> <artifactId>nifi-api</artifactId> </dependency> <dependency> <groupId>org.apache.nifi</groupId> <artifactId>nifi-utils</artifactId> </dependency> <dependency> <groupId>commons-io</groupId> <artifactId>commons-io</artifactId> </dependency> </dependencies> </project>

alfonz19 · ‎01-12-2018

Hi, we have issues with standard nifi processors, and have to fix them ourselves. I don't want to delve into details why, please take it as an axiom, imagine that I'm creating something from scratch instead. So I took nifi 1.4.0 archetype to generate project. I created some custom procesor on my own, which does not have any extra dependencies, and validated, that it works in nifi. All good. Now lets move on to nifi code. I took ValidateRecord file, and copy-pasted it into our project. I copy pasted also all dependencies from pom.xml module ValidateRecord is in. Our maven project and ValidateRecord maven project has now same parent, same dependencies. Both builds just fine. When I tried to deploy it, it fails, because of missing dependencies, for example on nifi-record. I build our project using mvn clean install in root project. Built nar file does NOT contain respective jar (nifi-record). When I build nifi, also nifi-standard-processors does not contain this jar. I'd consider it normal, since maven parent prescribes provided. We have this dependency in pom.xml: <dependency> <groupId>org.apache.nifi</groupId> <artifactId>nifi-record</artifactId> <scope>compile</scope> </dependency> (we added compile in despair, without it it does not work equally) Question one: how do you declare dependency in nifi/nar if you want it to be available and nar can be deployed?? Now into more wild stuff: if I run mvn nifi-nar:nar it will generate nar files in all 3 target dirs. In root dir with pom packaging it's pure nonsense, in processor's module, with jar packaging there is nar file, which contains blend of normal build and nar build, and it CONTAINS nifi-record library. And finally in packaging module there is nar without nifi-record; same as from mvn clean install. To conclude: provided, that I work with official archetype and official source code, while using identical dependencies definition, it's surprising that I'm not able to get working code. What am I doing wrong? (aside of copypasting, but I don't have any other choice) Thanks!

Online	Offline
Last Visited	‎01-19-2018 07:23 PM

Date Registered	‎01-12-2018 02:46 PM
Last Visited	‎01-19-2018 07:23 PM
Total Messages Posted	10
Total Kudos Received	1

Re: get flowfile for provenance event

get flowfile for provenance event

SplitJson behavior for non array input

Re: nifi exception handling / style

Re: nifi exception handling / style

nifi exception handling / style

Re: Please help to understand how to correctly bui...

Re: Please help to understand how to correctly bui...

Please help to understand how to correctly build N...