'Records Processed' missing for a processor

Question

On a processor, right click -> view status history....there is a data available by name 'Records Processed' in dropdown.

For one of the processor in my flow, this is not available.

What can be the reason for this? Is it possible that value was so high that system lost track somewhere?

MattWho · Accepted Answer

@manishg
The Record Reader and record Writer controller services are not responsible for tracking counts. This is handled within your custom processors code. I am not sure what your custom processor does and whether it makes sense to track "record processed" or some other record based stat, but you can look at the github code for other processors like PartitionRecord to see how RecordCount is handled.

If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

MattWho · Answer

@manishg
I am not clear, can you share the processor type where the expected data is not present?

The View Status History "Record Processed" will only appear on some component "Record" type processors (example: PartitionRecord processor). It would not exist as on processors that don't process "records".

If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.