Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Cannot push data to PostgreSQL using NiFi

Highlighted

Cannot push data to PostgreSQL using NiFi

I am trying to push data to Postgres using NiFi. I can see the data being populated in the insert query. Data is queued up on to the PutSQL processor but unable to push to the table. Please suggest what's wrong here. Thanks.

13448-putsql.jpg

13447-postgres-issue.jpg

4 REPLIES 4
Highlighted

Re: Cannot push data to PostgreSQL using NiFi

@spdvnz

I responded to another question from you. There is a configuration in "Settings" tab of "Configure" (right-click) on the queue that sets the limits for back pressure by default to 10,000 or 1 GB. One of your queues is backed-up. You need to stop some processors pushing that data to those queues that reached the limit and start some processors that will process and push data forward in order to unlock all these queues. My 2c. It happened to me.

Re: Cannot push data to PostgreSQL using NiFi

@Constantin Stanca

Thank you for your response for my another question. I want to understand why the data is being queued when the PutSQL processor is running? Even though all other processors are stopped the flow files are still in the queue. I don't think what you said would solve my issue.

Highlighted

Re: Cannot push data to PostgreSQL using NiFi

@spdvnz

The fact that the queue shows 9900 flow files that makes me to believe that the 100 that is the default value for the batch did not get committed to Postgres due to an error. What does the log indicate? Any error when trying to write to Postgres? The nifi-app.log is located usually under the path where your NiFi is installed.

Highlighted

Re: Cannot push data to PostgreSQL using NiFi

@Constantin Stanca

The queue was piling up without pushing data to Postgres so I stopped Setup SQL Query and the queue stopped at 9900. I see 10000 in the queue if I don't stop it. I don't get any error bulletin. Earlier I could see error/warn logs in nifi-app.log file. Now I have Ambari installed NiFi, I could only see heartbeat messages in nifi-app.log. I also didn't see any error in Ambari LogSearch.

Don't have an account?
Coming from Hortonworks? Activate your account here