Support Questions
Find answers, ask questions, and share your expertise

Problem adding duplicate object in Google Storage using PutGCSObject processor in Nifi

New Contributor

I am using Nifi to send data from Pub/Sub queue to Cloud storage. I'm using the ConsumeGCPubSub processor to fetch data from the queue and the PutGCSObject processor to add Cloud Storage in Nifi. But the PutGCSObject processor is sending duplicate data in Cloud Storage.

When I examine it, the number of data coming from the queue and exiting the PutGCSObject processor as success is the same, but I see that the data is written over and over again. When I looked into NiFi Data Provenance, I found that there are multiple data with the same FlowFile UUID.

I also see that this data has the same MD5 Hash code in its Cloud Storage records. What could be causing this and how can I fix it?

I double checked:

  • pub/sub messages is not duplicated.
  • When I send 30 piece of data, there are come exactly 30 pieces in Nifi
  • I checked my google storage have different data. But there was not..


; ;