Even with the standalone JAR, ExecuteSQL will not work with Hive due to the Hive JDBC driver not implementing some of the JDBC API calls made by ExecuteSQL. There is a Jira case to add Hive support: https://issues.apache.org/jira/browse/NIFI-981. There are some workarounds described in the following nifi-users email (from the archive): https://mail-archives.apache.org/mod_mbox/nifi-users/201601.mbox/%3C43A01CF2-2CC6-4A01-8836-86B9B3BEA31B@gmail.com%3E ... View more

If NiFi is running as a user that can change permissons in HDFS, you should be able to use the following properties in PutHDFS to do the chown/chmod: Permissions umask Remote Owner Remote Group ...according to the documentation: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.hadoop.PutHDFS/index.html ... View more

If the expression refers to attributes on a flow file, you will need to pass a reference to the flow file into evaluateAttributeExpressions: FlowFile flowFile = session.get(); jsonObject.put("hostname", context.getProperty(ADD_ATTRIBUTE).evaluateAttributeExpressions(flowFile).getValue()); If the property contains an attribute name (rather than an Expression containing an attribute name) and you want the value from the flow file: jsonObject.put("hostname", flowFile.getAttribute(context.getProperty(ADD_ATTRIBUTE).getValue())); If the attribute value itself contains Expression Language and you want to evaluate it, take a look at the following class: org.apache.nifi.attribute.expression.language.Query ... View more

I was able to generate and send JSON to InvokeHttp POST using a test endpoint (http://httpbin.org/post), it read the incoming flow file contents (the JSON) and correctly responded based on the input file. Does the InvokeHttp response make sense for what you intend to send via POST? If not, you may need to (if you haven't already) set the "mime.type" attribute to "application/json" before sending the flow file to InvokeHttp. This can be done with the UpdateAttribute processor (see my example template below). If GetFile still doesn't seem to work, perhaps try ListFile followed by FetchFile. Having said that, I wouldn't think this would make a difference, I suspect there's something else in the flowfile content and/or attributes (or a lack thereof) that causes InvokeHttp to not respond as intended. My template is available as a Gist (here). ... View more

I don't think PostHTTP outputs the response as a flow file. If you are looking for the response to be in a flow file, try InvokeHttp (with HTTP Method property set to "POST") instead and send its "response" relationship to your PutHDFS processor. ... View more

For replacing contents of an incoming flowfile, the write(FlowFile, StreamCallback) method of ProcessSession is probably your best bet. There is a corresponding section of the NiFi Developer's Guide. There is an example of this in the ReplaceText processor code under the AlwaysReplace strategy. As an alternative to creating a full-fledged custom processor, as of NiFi 0.5.0 there are the InvokeScriptedProcessor and ExecuteScript processors. You can use InvokeScriptedProcessor to use a scripting language like Groovy to provide a Processor implementation, which avoids the need for building and deploying your own NARs. If you don't need a full processor and are just looking to manipulate the flowfile content, you could use the ExecuteScript processor, which allows easy access to incoming flowfiles and such. I've written a blog post on using ExecuteScript (with Groovy) to replace flowfile contents: http://funnifi.blogspot.com/2016/02/executescript-processor-replacing-flow.html Regards, Matt ... View more

There sure is! I am currently working on NIFI-1275 to implement FetchElasticsearch and PutElasticsearch, the former uses the Get API and the latter uses the Bulk Insert API. Are there any particular features you're interested in having supported? We are hoping to offer support for secure Elasticsearch instances, but may not be able to provide out-of-the-box support due to licensing/distribution issues. Querying (aka the Search API) is not currently planned under NIFI-1275 but the InvokeHttp processor can perform the desired query, then once the document IDs are retrieved, FetchElasticsearch can be used to get the documents' contents. ... View more

I see at least two use cases: I know <non-Java language here> but I want to write a processor for NiFi. There is no (current) processor or workflow that takes care of my needs, so I’d like to write code to fill in the blanks( basically the body of an onTrigger() call ) Hot-reload is common to both. Adding new languages on-the-fly is not (IMHO). For "Invocable" script engines (like Jython) that allow you to compile an implementor of Processor, would be nice to call into those scripts and treat them like first-class Processors. For other scripts / script engines, it would be cool to treat the script body as the onTrigger() API call -- to be executed when scheduled and when a FlowFile is available. ... View more