PutMarklogic - fails to load the data when the queue size is 10000

[#1]

Hi ,

If the  queue size is less than 1000 json records then putMarklogic processor successfully able to load the document into marklogic db, where as if more than say 5K records it is getting failed 

 Could you please suggest do i need to updated any batch size ,yield timing /schedule timing  

[#2]

could you please assist me on this request 

[#3]

+1 on @cotopaul and @steven-matison being rockstars in the community.
@swanifi 
The PutMagicLogic processor is not included out of the box with Apache NiFi releases. It is a custom component build by other members in the Apache community (https://github.com/marklogic). You may find better help by filing an Apache Jira within the marklogic project here:
https://github.com/marklogic/nifi/issues/new

Aside from reading the MarkLogic documentation, I would be of little help here:
https://marklogic.github.io/nifi/step-by-step

As @steven-matison mentioned you may be able to limit the size of the queue on the connection feeding the PutMarklogic processor to avoid exceeding 5000 (if that happens to be some limit on this processors capability).  

https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#settings
The Back Pressure Object Threshold is a soft limit for the connection.  As a soft limit it simply, if the connection is >= the configured threshold, the processor component feeding that connection will not get scheduled to execute until that threshold drops back below the configured threshold.  Some components process FlowFiles in batches. Some ingest processors like ListSFTP, ListFile, etc. have potential to generate a lot of FlowFiles in a single execution.  If your connection source is one that produce a lot of FlowFiles in a single execution, you could add a processor in between to have better control over the connection feeding the Put MagicProcessor (example: UpdateAttribute, controlRate, etc.)

While this does not solve your issue with this processor itself, it may help you move forward with its existing behavior.

If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.

Thank you,

Matt

[#4]

@swanifi, Welcome to our community! To help you get the best possible answer, I have tagged in our Nifi experts @MattWho @ckumar @cotopaul @SAMSAL who may be able to assist you further.

Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.

[#5]

Hi @swanifi,

I would start by uploading the full error stack and not a photo of a partial error, taken with your phone. Afterwards, I would tell you that the mentioned processor, based on my knowledge, is not an official processor, belonging to Apache NiFi or Cloudera NiFi stack. This means that in most cases, you will have to discuss this topic directly with the developer of that custom made processor. In addition to this, I would double check the MarkLogic DB for any restrictions in term of connectivity (restrict the number of inserts via API to a number of X items)

[#6]

 @cotopaul   - Thanks for being such a community superstar!!

@swanifi in addition to above, In the future please be sure to include screen shot of the processor property and configurations.

One thing that i think you could achieve is some type of an upstream gate on the flow files, so that they are released in blocks that satisfy the expectation of 1000 records per operation.    I am not aware if this is configureable within the processor.   

[#7]

@swanifi, Have any of the replies helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.