For approach #1, you could use the FlattenJson processor, you'll likely want to set the Separator property to "_" rather than the default "." since Hive adds the table name to each column in a ResultSet. For approach #2, you could have a single column table (column of type String), then you'd query it with get_json_object (example here). Alternatively if you can map all the types (including the complex types like array, list, struct, etc.) to a Hive table definition, you could use a JSON SerDe to write the data (example here). ... View more

Fetch Size is a hint to the driver that says "when I ask for a row, please send X rows over, if I need more rows I will iterate through them before asking you to send more rows". If Fetch Size was 1, the driver would send a network request to the DB, the DB would send a single row back, it would be processed by the client, then for the next row another network request would be sent, and so on. This is very inefficient as the network overhead would overtake the amount of data being passed (especially for tables with a small number of rows). On the other hand, if the entire table was sent at the first request, the client could run out of memory, especially for tables with a very large number of rows. As a tuning parameter, many drivers allow the client to specify the Fetch Size in order to select the appropriate number of rows to send during a network request/response. For example I think the Hive driver's default is 50 rows, or at least it used to be. Maximum Number of Fragments is an advanced user property usually only used for debugging. It basically stops fetching rows after a certain number have been processed, even if the result set has more rows available. For example, if Max Rows Per Flow File were set to 100, and Max Number of Fragments were set to 1, and a query returned 101 rows, then only 1 flow file (aka "fragment" in this case) would be sent, containing the first 100 rows. The 101th row would not be processed or sent in any flow file. Re-running the processor would have the exact same effect, and thus the 101th row is "unreachable". Maximum Number of Fragments was added to let the user avoid memory problems for large tables, as it would put an upper bound on the number of rows that would be processed. As mentioned, certain rows would not be available, but the processor wouldn't run out of memory either. Hope this helps, please let me know if you'd like more clarification. ... View more

The schema is a valid Avro schema, but the problem is that the CSVReader is taking only the second field (a double) and trying to convert it into an array. It does not realize you'd like to convert the remaining column values into an array. This seems like a decent improvement for the CSVReader, I'll think it over and perhaps write an Improvement Jira for it (or feel free to write one yourself). Since you want JSON as the target, you can use JoltTransformJSON to get the output you want. If it were an arbitrary output format, you could use UpdateRecord, but you'd have 35 or so user-defined fields to put the individual column values into the array. In any case, you can use this schema to parse the column names (whether or not you have a header line, just make sure you skip any header line if using this schema): { "type": "record", "namespace": "testavro.schema", "name": "test", "fields": [ { "name": "timestamp", "type": {"type": "long","logicalType": "timestamp-millis"} }, { "name": "tupvalue0", "type" : "double" }, { "name": "tupvalue1", "type" : "double" }, { "name": "tupvalue2", "type" : "double" }, { "name": "tupvalue3", "type" : "double" }, { "name": "tupvalue4", "type" : "double" }, { "name": "tupvalue5", "type" : "double" }, { "name": "tupvalue6", "type" : "double" }, { "name": "tupvalue7", "type" : "double" }, { "name": "tupvalue8", "type" : "double" }, { "name": "tupvalue9", "type" : "double" }, { "name": "tupvalue10", "type" : "double" }, { "name": "tupvalue11", "type" : "double" }, { "name": "tupvalue12", "type" : "double" }, { "name": "tupvalue13", "type" : "double" }, { "name": "tupvalue14", "type" : "double" }, { "name": "tupvalue15", "type" : "double" }, { "name": "tupvalue16", "type" : "double" }, { "name": "tupvalue17", "type" : "double" }, { "name": "tupvalue18", "type" : "double" }, { "name": "tupvalue19", "type" : "double" }, { "name": "tupvalue20", "type" : "double" }, { "name": "tupvalue21", "type" : "double" }, { "name": "tupvalue22", "type" : "double" }, { "name": "tupvalue23", "type" : "double" }, { "name": "tupvalue24", "type" : "double" }, { "name": "tupvalue25", "type" : "double" }, { "name": "tupvalue26", "type" : "double" }, { "name": "tupvalue27", "type" : "double" }, { "name": "tupvalue28", "type" : "double" }, { "name": "tupvalue29", "type" : "double" }, { "name": "tupvalue30", "type" : "double" }, { "name": "tupvalue31", "type" : "double" }, { "name": "tupvalue32", "type" : "double" }, { "name": "tupvalue33", "type" : "double" }, { "name": "tupvalue34", "type" : "double" } ] } Your JSONRecordSetWriter can inherit the schema or use the same one from the registry. Now you can send the output to either UpdateRecord or (for this case) JoltTransformJSON, the following is a Chain spec you can use to put the tupvalues into an array: [ { "operation": "shift", "spec": { "*": { "timestamp": "[&1].timestamp", "tupvalue*": "[&1].tupvalue[&(0,1)]" } } } ] This should give you the desired output and works on single lines of CSV or (more efficiently for) full CSV files. ... View more