Hi ,   Thanks for the reply . As i mentioned that i have run the insert ino tableA select * from tableB (Assume Delta calculation query). Yes , we have managed table in hive and hive version is 3.0 where ACID are enabled by default .We have created the Hive managed table through HUE and executed the delta query via NiFi PutHQL processor and we got the error intermittently .     ... View more

@MattWho    In continuation of above logic and i implemented and it is working fine . I have three nodes nifi cluster where a,b,c .Here a node is primary node always . If a goes out of space , we will unzip the file from another node and distributes to b,c nodes and we push the file to HDFS folder and we capture success and failure message( in Updateattribute processor)  as mentioned in the posts above . After that in mergerecord process , the queue is waiting for sometime and merging happens after 10mins around . I have a dependency of the merging processor as i have to share the status of flowfile to another API which is requesting for it. How we can speed up the MergeRecord processor here . If all three nodes have good amount of memory and the flowfiles pushed to HDFS and status updates (through UpdateProcessorattribute) and merging happens very quickly as expected in single node. But we have nodes(b,c) and 'a' primary node is out due to space issue in device .The processor unpack the .zip file then distribute to nodes tp put into HDFS .As we capture each success/failure status of PutHDFS (using updateattribute processor)  and  mergerecord is taking sometime to merge the status . Is this because of other two nodes processing it slowly or some other reason for that.   I have attached screenshot of the partial flow . ... View more

Hi  Faerballert  Thanks for helping . I am able to get entire content into attribute . I have one doubt that i have attribute name called "JSONValue" so that extracted value should be in this name but the flowfile content saves into the attribute  "JSONValue.0 "  as supposed to create value in JSONValue attribute . Please see screen-print for the same . I had to enable "Include Capture Group 0" as True otherwise the Regex not accepting and expecting value 0 to40 other than false.     ... View more

@MattWho    This is Amazing and solution working well. ... View more

Hi , @MattWho Thanks a lot for clarifying and provide solution for Merging contents . I have implemented as per your guidelines and the merge content working fine But i am unable to update overall status of file if it is error and message as well. Please find my status flow in detail.    1.Your flow starts with a single zip file? --- I get meta file which contains .zip file to be fetched . for exmaple whosale.meta file which contain .zip file having 3 or 4 csv files .  2. You unzip that file to produce numerous output FlowFiles? --- Yes , I unzip the file . 3. You use load balanced connections to distribute all the produced FlowFiles across all nodes in your cluster? --- Yes , i do enable round robin load balancing for the flow files which was unzip in step 2 and push the flowfiles into HDFS . 4. Then you modify content of FlowFile using convertAttribitesToJson processor (Destination=flowfile-content)?  Looks like you route the "success" relationship twice from this processor which means you have cloned your FlowFiles.  Why?  --- Initially i was trying convert attributes into JSON flowfile. I have to capature success and failure status of the processor .So i have avoided not clone the success relationship twice.       6. Where do the "failure" FlowFiles get introduced in to this dataflow? --- If i encounter "failure" in process of pushing files to HDFS like permission denied or folder not available . Then i will add attribute with some custom messages and send to merge content . And i have also dirceting success status with some custom message of the final process success status in attribute to replacetext followed with Mergecontent .   Status flow      HDFSError attribute :      HDFS Successattribute :   After merge content i do use Replacetext processor as suggested to add overall status of the file but somehow the status (error) and message is not captured .   ReplaceTextProc to construct over all status :       The final file will contain like below    Ingest_flow.zip : Success :  test1.csv : Success : File artikelgroep_info.csv pused to PreRAW Zone test2.csv : Success : File demo.csv pused to PreRAW Zone test3.csv : Success : File work11.csv pused to PreRAW Zone   i could not populate over all message AND if any file error then i could not capture status " Failed" and with error in overall status of the .zip file .Could you please help on this      Thanks ... View more