@Rahoul A You can use Control Rate processor before PutHDFS processor and configure Control Rate Processor to release flowfile for the desired time like 1 min ..etc. if you need to append data to the file then we need to make sure we are having same filename to get same filename every time we can use Update attribute processor to change the filename and in PutHDFS processor we need to configure the below property Conflict Resolution Strategy append //if processor finds same filename it appends the data to the file. Control Rate Processor configs:- By using these configurations we are releasing 1 flowfile for every one minute so at any point of time we are going to have one node write/append data to the file. Flow:- other Processors --> ControlRate Processor --> PutHDFS ... View more

@Laurie McIntosh If you are having more than one file to process then use Merge Content processor after GetFile processor,Merge Content processor merges more than one file into one file. Flow:- Get File --> MergeContent -->Truncate Table --> insert csv into Table --> clean up by using merge content processor we are processing one file at a time even though you are having more than one file. PutDatabaseRecord processor(all record based processors) are pretty powerful in NiFi which can handle millions of records. Please refer to below links to know how to configure merge content processor https://community.hortonworks.com/questions/64337/apache-nifi-merge-content.html https://community.hortonworks.com/questions/161827/mergeprocessor-nifi-using-the-correlation-attribut.html https://community.hortonworks.com/questions/149047/nifi-how-to-handle-with-mergecontent-processor.html in addition there are wait and notify processors in NiFi, which Routes incoming FlowFiles to the 'wait' relationship until a matching release signal is stored in the distributed cache from a corresponding Notify processor. When a matching release signal is identified, a waiting FlowFile is routed to the 'success' relationship. http://ijokarumawak.github.io/nifi/2017/02/02/nifi-notify-batch/ ... View more

@Mahendra Hegde Increase the below properties to at least 10,15 secs and try to run the processor again because as you have configured the processor with 3,5 secs now which is pretty low value for timeouts, there is a chance when you are making a call the client probably client is busy with handling other requests and if you are not able to connect to the client with in 3 secs then the request will be routed to NoRetry relation. Connection Timeout 10 secs Max wait time for connection to remote service. Read Timeout 15 secs Max wait time for response from remote service. The flowfile will be transferred to NoRetry relationship when there are 1xx, 3xx, 4xx status codes as you can view the attributes in the flowfiles that routed to NoRetry relation you will have the status code and error that you are getting when the call made to the service. RestApi Error codes: CATEGORY DESCRIPTION 1xx: Informational Communicates transfer protocol-level information. 2xx: Success Indicates that the client’s request was accepted successfully. 3xx: Redirection Indicates that the client must take some additional action in order to complete their request. 4xx: Client Error This category of error status codes points the finger at clients. 5xx: Server Error The server takes responsibility for these error status codes. Please refer to this link for more details about restapi error codes. if you still having issues with the above configs and you are getting responses from invoke http processor after couple of retries then try to increase the values again ,try to make call again. ... View more

@Laurie McIntosh If you are using NiFi1.5+ then you can use PutSQL processor with SQL Statement property as your Truncate statement,as this processor won't change the contents of flowfile, so we can use PutDatabaseRecord processor to prepare SQL statements and finally use PutSQL processor for post insert. jira addressing SQL statement property in PutSQL processor, https://issues.apache.org/jira/browse/NIFI-4522 Flow:- If you are running Prior version of NiFi1.5 then use executescript processor to run truncate statement on the target database and then use PutDatabaseRecord and PutSQL processors. Please refer to this link for more details. . If the Answer helped to resolve your issue, Click on Accept button below to accept the answer, That would be great help to Community users to find solution quickly for these kind of issues. ... View more

Thanks @jwitt, i missed the first step of splitting, i thought its new flowfile for each message. ... View more

@Saikrishna Tarapareddy Step1:Split Text processor:- Configure split text processor with Line Split Count 1 (or) Use Split Content processor with Byte Sequence Format Text Byte Sequence shift+enter Use splits relation from the above processors feed to merge content processor. Splits relation will have each line of your json file as a new flowfile content then you can use Merge Content processor to merge the json messages and make them as an array of json. Step2:Merge Content Processor configs:- Delimiter Strategy Text Header [ Footer ] Demarcator , Configure the processor for your desired Minimum number of entries and use Max Bin age as wild card. Please refer to below link for merge content processor configs. https://community.hortonworks.com/questions/64337/apache-nifi-merge-content.html https://community.hortonworks.com/questions/161827/mergeprocessor-nifi-using-the-correlation-attribut.html https://community.hortonworks.com/questions/149047/nifi-how-to-handle-with-mergecontent-processor.html ... View more

@Nikhil R {"component":{"id":'controllerID',"name":"DBCPConnectionPool", 'state': 'DISABLED'},"revision":{"clientId":client,"version":11}} Change the quotes in your json payload keep everything in double quotes don't use single quotes. {"revision":{"clientId":"af22b3ab-0162-1000-9b81-cf7b337dfc47","version":1},"component":{"id":"c8b1c581-47f0-3773-e7a5-3eb739246f99",'state':'DISABLED'}} ... View more

@Nikhil R In your pay load you have to include your client id number for the controllerservice. Example:- I'm having a controller service in my nifi instance and we need to get the clientId for the controller service. Step1:How to find clientId value? To find out client id and version number use Developer tools (chrome, firefox etc) and perform any action (start, stop ...etc) for the controller service and look at the calls made for the Controller service id. for reference take a look in the below screenshot Click on Network 2.In filter keep your controller service id 3.Click on Response then you can find clientid,version Once you get all the values prepare your curl command to stop Controller Service. As my controllerservice id is c8b1c581-47f0-3773-e7a5-3eb739246f99 so i filtered out only this specific id and then go to any of the Put methods and go to Params tab on right side then you can get the clientId associated with the controllerservice. Step2:Prepare PayLoad Now include your clientid value in your payload, from the above screenshot my clientId value is af22b3ab-0162-1000-9b81-cf7b337dfc47 so i have included the same clientId in my payload and included my componentid i.e our controller service id {"revision":{"clientId":"af22b3ab-0162-1000-9b81-cf7b337dfc47","version":1},"component":{"id":"c8b1c581-47f0-3773-e7a5-3eb739246f99","state":"DISABLED"}} Step3:Prepare Curl api call:- Now we need to include payload and end point in one curl call. Use put method and the state as DISABLED(as we need to disable the controller service) in curl call. Curl api call:- curl -i -X PUT -H 'Content-Type:application/json' -d '{"revision":{"clientId":"af22b3ab-0162-1000-9b81-cf7b337dfc47","version":1},"component":{"id":"c8b1c581-47f0-3773-e7a5-3eb739246f99","state":"DISABLED"}}' http://localhost:9090/nifi-api/controller-services/c8b1c581-47f0-3773-e7a5-3eb739246f99 Once the api call is success then you can receive 200 response code and the controller service will be disabled. For your case change the the clientid and controllerserviceid make rest api call to disable the controller service. . If the Answer helped to resolve your issue, Click on Accept button below to accept the answer, That would be great help to Community users to find solution quickly for these kind of issues. ... View more