I'm not a Hive expert but I did author the original PutHive3Streaming processor for NiFi. My recommendation is setting Records Per Transaction greater than the number of records in a FlowFile (unless we are talking about super-huge files), and transactions per batch to 1. This makes the transaction semantics similar to how NiFi FlowFile sessions work (rollback, failure, success, e.g.). If the number of records is huge and is causing throughput problems, try dividing that number by 100 and making transactions per batch 100. When you multiply the two numbers together it should be greater than the total number of records in the FlowFile in order to avoid overhead with the Hive Metastore by requesting a large number of batches/transactions. ... View more

The operation to add an attribute to a FlowFile is on the ProcessSession object not the FlowFile itself (so the session can keep track of changes).  Try the following instead:   session.putAttribute(destFlowFile, , "logMsg", "Testing Msg") session.putAllAttributes(destFlowFile, backupAttributes) ... View more

You can refer to the "Fields" output field explicitly instead of needing another shift:   [ { "operation": "shift", "spec": { "*": { "CUST_AC_NO": "[&1].ExternalSystemIdentifier", "BRANCH_CODE": "[&1].Fields.FLD0001", "CUST_NO": "[&1].Fields.FLD0002", "AC_DESC": "[&1].Fields.FLD0003" } } }, { "operation": "default", "spec": { "*": { "InstitutionId": "1" } } } ] ... View more

If I am reading your use case correctly, I think you're looking for what the ForkRecord processor does; it allows you to fork a (usually single) record into multiple records based on a Record Path (similar to JSONPath but different syntax and expressiveness), possibly keeping the "root" elements common to each outgoing record. ... View more

If it works in NiFi 1.5 and not in 1.9.2, then the DBCP libraries probably got updated to DBCP 2, where the latter will call isValid() on the connection if a validation query is not set (see https://sourceforge.net/p/jtds/discussion/104388/thread/bbfbcf24/). Have you set the Validation Query property on the DatabaseConnectionPool? If not, set it to "SELECT 1" and try again, hopefully that will work. If that doesn't, you can try adding a user-defined property called "validationQuery" and set it to "SELECT 1". That should add it as a property on the JDBC connection itself. Another workaround would be to use the official SQL Server driver (although I assume you chose jTDS on purpose). ... View more

In nifi-assembly/target you'll find the built system as you mention, including a "conf" folder that contains (among other things) a file called bootstrap.conf. In that file there's a commented out JVM property to enable attachment by a debugger (the preceding line says "Enable Remote Debugging". When you uncomment that argument and start NiFi, it will listen on port 8000 for a debugger to attach. You can then attach a debugger from your IDE (Eclipse, NetBeans, IntelliJ, etc.). You can change the port and/or set "suspend=y" if you want it to wait until a debugger is attached before continuing startup, the latter is helpful if you are debugging something early in the startup sequence. Otherwise you can wait for NiFi to finish starting up and then attach whenever you like. ... View more

You were so close! By using the [] syntax it just adds to the outgoing array, but you wanted to associate them with the same index, namely the one matched by the * "above" the fields. Put #2 inside your braces (#2 is a reference to the array index you're iterating over, "two levels up" from where you are in the spec): [{     "operation": "shift",     "spec": {       "nummer": "Nummer",       "table": {         "*": {           "zn": "Positionen.[#2].ZeileNr",           "datum": "Positionen.[#2].Datum"         }       }     } }] ... View more