What version of NiFi are you using? Is the "value" column in your database table a String or a Float/Double? What processor(s) are you using to read from the database? If using ExecuteSQL, could you do something like the following? SELECT metric, CAST(value AS DOUBLE) AS value, timestamp, tags from myTable Alternatively, as of NiFi 1.2.0 (HDF 3.0) you can use the JoltTransformJSON processor to do type conversion (see an example here). Also if you know what the schema is supposed to be, you could use ConvertRecord with a JsonRecordSetWriter which is associated with the "correct" schema. The reader can be an AvroReader which uses the Embedded Schema. ... View more

It appears you want to set the destination path to the value of type, followed by the value of id, followed by data.txt, and in the content of that file you want the single-element JSON array containing the object that provided the values. If that is the case: As of NiFi 1.3.0, there is a PartitionRecord processor which will do most of what you want. You can create a JsonReader using the following example schema: {"type":"record","name":"test","namespace":"nifi", "fields": [ {"name":"type","type":"string"}, {"name":"id","type":"string"}, {"name":"content","type":"string"} ] } You can also create a JsonRecordSetWriter that inherits the schema (as of NiFi 1.4.0) or uses the same one (prior to NiFi 1.4.0). Then in PartitionRecord you would create two user-defined properties, say record.type and record.id, configured as follows: Given your example data, you will get 4 flow files, each containing the data from the 4 groups you mention above. Additionally you have record.type and record.id attributes on those flow files. You can route them to UpdateAttribute where you set filename to data.txt and absolute.path to /${type}/${id}. Then you can send them to PutHDFS where you set the Directory to ${absolute.path}. ... View more

I'm not familiar with the innards of either Groovy or Jython, but I am guessing that Jython is slower for the following reasons: 1) Groovy was built "for the JVM" and leverages/integrates with Java more cleanly 2) Jython is an implementation of Python for the JVM. Looking briefly at the code, it appears to go back and forth between the Java and Python idioms, so it is more "emulated" than Groovy. 3) Apache Groovy has a large, very active community that consistently works to improve the performance of the code, both compiled and interpreted. In my own experience, Groovy and Javascript (Nashorn) perform much better in the scripted processors than Jython or JRuby. If you choose Jython, there are still a couple of things you can do to improve performance: - Use InvokeScriptedProcessor (ISP) instead of ExecuteScript. ISP is faster because it only loads the script once, then invokes methods on it, rather than ExecuteScript which evaluates the script each time. I have an ISP template in Jython which should make porting your ExecuteScript code easier. - Use ExecuteStreamCommand with command-line Python instead. You won't have the flexibility of accessing attributes, processor state, etc. but if you're just transforming content you should find ExecuteStreamCommand with Python faster. - No matter which language you choose, you can often improve performance if you use session.get(int) instead of session.get(). That way if there are a lot of flow files in the queue, you could call session.get(1000) or something, and process up to 1000 flow files per execution. If your script has a lot of overhead, you may find handling multiple flow files per execution can significantly improve performance. ... View more

What do you mean by "add to this JSON a file that I get from FetchFTP"? Is the file you're fetching a JSON file, and you want to add fields to it? Are you Base64 encoding just the JSON from the attributes or the entire file after adding to it? If the incoming file (from FTP) is JSON, and you can get your attributes added to that flow file, then (as of NiFi 1.2.0 / HDF 3.0) you can use JoltTransformJSON to inject your individual attributes as fields into your JSON document (instead of AttributesToJSON). If you have too many attributes for that, your options are a bit more limited. In NiFi 1.3.0, you can use UpdateRecord to add the JSON from an attribute into a field in the other JSON document. You can also do this manually with ReplaceText. However one of the two JSON objects must be in an attribute. Whichever of the two (from AttributesToJSON or FetchFTP) is smaller, you can get that object first and use ExtractText to put the whole thing into an attribute. Note that attributes have limited size and introduce more memory usage, so beware of large JSON objects in attributes. However if one of them fits in a attribute, then you can use the UpdateRecord or ReplaceText processor as described. If you need to just encode one of the JSON objects, then if it is in an attribute you can use UpdateAttribute with the base64Encode Expression Language function, or if it is in content you can use the Base64EncodeContent processor. ... View more