Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

NiFi RunMongoAggregation error: "exceeded memory limit for $group, but didn't allow external sort"

Highlighted

NiFi RunMongoAggregation error: "exceeded memory limit for $group, but didn't allow external sort"

New Contributor

Hi All,

I used a RunMongoAggregation processor to aggregate MongoDB collection and expected it to return around 10k+ documents. Upon running of the processor, I encountered following error: "exceeded memory limit for $group, but didn't allow external sort".

Following is the Query I used:

[   
    {"$group": {   
     "_id": "$meta_entity_id", 
     "meta_timestamp": {"$last": "$meta_timestamp"},
     "meta_start_date": {"$last": "$meta_start_date"},
     "meta_end_date": {"$last": "$meta_end_date"},
     "meta_version": {"$last": "$meta_version"},
     "meta_operation": {"$last": "$meta_operation"},
     "nric": {"$last": "$nric"},
     "name": {"$last": "$name"},
     }
    }
]

Based on MongoDB documentation, $group might have some limitation:

The $group stage has a limit of 100 megabytes of RAM. By default, if the stage exceeds this limit, $group will produce an error. However, to allow for the handling of large datasets, set the allowDiskUse option to true to enable $group operations to write to temporary files.

I tried to add allowDiskUse option, but it was not working i.e. same issue

[   
    {"$group": {   
     "_id": "$meta_entity_id", 
     .
     .
     .
     }
    }
], { allowDiskUse: true }

Any idea how to fix?

Thanks

1 REPLY 1

Re: NiFi RunMongoAggregation error: "exceeded memory limit for $group, but didn't allow external sort"

New Contributor

I am also experiencing the issue described while running an aggregation and have no luck getting the { allowDiskUse: true } option to be picked up.


Did you manage to resolve this?