I am trying to push data to Postgres using NiFi. I can see the data being populated in the insert query. Data is queued up on to the PutSQL processor but unable to push to the table. Please suggest what's wrong here. Thanks.
I responded to another question from you. There is a configuration in "Settings" tab of "Configure" (right-click) on the queue that sets the limits for back pressure by default to 10,000 or 1 GB. One of your queues is backed-up. You need to stop some processors pushing that data to those queues that reached the limit and start some processors that will process and push data forward in order to unlock all these queues. My 2c. It happened to me.
Thank you for your response for my another question. I want to understand why the data is being queued when the PutSQL processor is running? Even though all other processors are stopped the flow files are still in the queue. I don't think what you said would solve my issue.
The fact that the queue shows 9900 flow files that makes me to believe that the 100 that is the default value for the batch did not get committed to Postgres due to an error. What does the log indicate? Any error when trying to write to Postgres? The nifi-app.log is located usually under the path where your NiFi is installed.
The queue was piling up without pushing data to Postgres so I stopped Setup SQL Query and the queue stopped at 9900. I see 10000 in the queue if I don't stop it. I don't get any error bulletin. Earlier I could see error/warn logs in nifi-app.log file. Now I have Ambari installed NiFi, I could only see heartbeat messages in nifi-app.log. I also didn't see any error in Ambari LogSearch.