Yes, you can use ListFile -> FetchFile. ListFile will keep track of which files it has read, so it will only list those files that it has not seen before. FetchFile will get the contents of the files passed in. Together they work like GetFile except ListFile keeps track of the files it has read. ... View more

After your FetchSFTP, the bar-delimited content will be in the content of the flow file, not the attributes. That is followed by an AttributesToJson processor which will overwrite the flow file content with a JSON document containing attributes such as sftp.remote.host, sftp.remote.port, etc. (see the doc for AttributesToJson). I think you may want a SplitText processor after your FetchSFTP processor, to create one flow file for each line in your file. Then you could have an ExtractText processor which could use a regex (with grouping) to create attributes such as column.1, column.2, etc. Then your ReplaceText can use those attributes. ... View more

That class is part of the write-ahead log stuff in nifi-commons. I think there was a recent update to its structure. Now that Apache NiFi 1.0.0 has been released, you should use that instead of the 1.0.0-BETA (which i see in your logs is the version you're on) ... View more

The output from one processor to another is a flow file, which consists of a map of attributes (key/value pairs) and a payload of bytes as the flow file content. The content could be raw binary data (an image, for example) or a text file in any format (JSON, XML, CSV to name a few). The content and attributes of a flow file are manipulated by the processors in different ways, the documentation for each processor will describe what attributes it reads and/or writes, as well as what operations it may perform on the processor. For example, the UpdateAttribute processor allows you to add attributes to (or delete them from) incoming flow files. Another example is SplitJson, which expects incoming flow files to have a JSON object as the flow file content, and then you configure the processor with a JSONPath expression pointing at an array within the object. Then the processor will split the original JSON object into individual JSON objects and send out flow files for each element in the array to the "split" relationship. It also sends the original incoming flow file to the "original" relationship. You can add connections between processors for the relationship(s) defined by the source processor. I encourage you to read the Overview and Getting Started guides for more information on the concepts of NiFi (Flow Files, Processors, Connections, etc.). If you are looking for working examples, there is a set of templates available on the NiFi Wiki. ... View more

Although the script engine reports its name as "python", it is actually Jython, which can only use pure Python modules, not native modules like numpy/scipy. If this is needed, consider ExecuteProcess or (if you have incoming flow files) ExecuteStreamCommand which can execute the command-line python. ... View more