To @Artem Ervits comment, if the Java class is available in the class loader (either by being part of the NiFi Core or being specified as a JAR in the ExecuteScript's Module Directory property), then you can call it directly, you don't need reflection. If I'm not answering your question, can you provide more details? I'm happy to help 🙂 ... View more

As of NiFi 0.6.0, there is a processor called QueryDatabaseTable that does something like this: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.QueryDatabaseTable/index.html You can enter the columns for which you'd like the processor to keep track of, it will store the maximum value it has "seen" so far for each of the specified columns. When the processor runs, it will generate a SQL query to return rows whose values in those columns are greater than the observed maximum values. So if you have a unique (and increasing) ID field, for example, you can specify that in the QueryDatabaseTable processor, then the first execution will return all rows, and subsequent executions will not return any rows until one (or more) have an value for ID greater than the max. This can be used to do incremental fetching like Sqoop does for only added/updated rows, using columns that contain timestamps, IDs, or other values that are increasing. ... View more

Even with the standalone JAR, ExecuteSQL will not work with Hive due to the Hive JDBC driver not implementing some of the JDBC API calls made by ExecuteSQL. There is a Jira case to add Hive support: https://issues.apache.org/jira/browse/NIFI-981. There are some workarounds described in the following nifi-users email (from the archive): https://mail-archives.apache.org/mod_mbox/nifi-users/201601.mbox/%3C43A01CF2-2CC6-4A01-8836-86B9B3BEA31B@gmail.com%3E ... View more

If NiFi is running as a user that can change permissons in HDFS, you should be able to use the following properties in PutHDFS to do the chown/chmod: Permissions umask Remote Owner Remote Group ...according to the documentation: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.hadoop.PutHDFS/index.html ... View more

If the expression refers to attributes on a flow file, you will need to pass a reference to the flow file into evaluateAttributeExpressions: FlowFile flowFile = session.get(); jsonObject.put("hostname", context.getProperty(ADD_ATTRIBUTE).evaluateAttributeExpressions(flowFile).getValue()); If the property contains an attribute name (rather than an Expression containing an attribute name) and you want the value from the flow file: jsonObject.put("hostname", flowFile.getAttribute(context.getProperty(ADD_ATTRIBUTE).getValue())); If the attribute value itself contains Expression Language and you want to evaluate it, take a look at the following class: org.apache.nifi.attribute.expression.language.Query ... View more

I was able to generate and send JSON to InvokeHttp POST using a test endpoint (http://httpbin.org/post), it read the incoming flow file contents (the JSON) and correctly responded based on the input file. Does the InvokeHttp response make sense for what you intend to send via POST? If not, you may need to (if you haven't already) set the "mime.type" attribute to "application/json" before sending the flow file to InvokeHttp. This can be done with the UpdateAttribute processor (see my example template below). If GetFile still doesn't seem to work, perhaps try ListFile followed by FetchFile. Having said that, I wouldn't think this would make a difference, I suspect there's something else in the flowfile content and/or attributes (or a lack thereof) that causes InvokeHttp to not respond as intended. My template is available as a Gist (here). ... View more

I don't think PostHTTP outputs the response as a flow file. If you are looking for the response to be in a flow file, try InvokeHttp (with HTTP Method property set to "POST") instead and send its "response" relationship to your PutHDFS processor. ... View more

For replacing contents of an incoming flowfile, the write(FlowFile, StreamCallback) method of ProcessSession is probably your best bet. There is a corresponding section of the NiFi Developer's Guide. There is an example of this in the ReplaceText processor code under the AlwaysReplace strategy. As an alternative to creating a full-fledged custom processor, as of NiFi 0.5.0 there are the InvokeScriptedProcessor and ExecuteScript processors. You can use InvokeScriptedProcessor to use a scripting language like Groovy to provide a Processor implementation, which avoids the need for building and deploying your own NARs. If you don't need a full processor and are just looking to manipulate the flowfile content, you could use the ExecuteScript processor, which allows easy access to incoming flowfiles and such. I've written a blog post on using ExecuteScript (with Groovy) to replace flowfile contents: http://funnifi.blogspot.com/2016/02/executescript-processor-replacing-flow.html Regards, Matt ... View more