Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

nifi 1.9 throwing error on update record while translating json to avro and the error does not explain much

nifi 1.9 throwing error on update record while translating json to avro and the error does not explain much

New Contributor

nifi_error.png

 So basically the steps we followed here :

1) reading json 

2) splitting the json

3) update record reads json and translate it to avro where we are getting this error. On this note we have the same code on nifi version 1.7 and that does not thrpw any sort of errors.

 

Could anyone give me some hint where we are going on ? Is it kind of plugin issues we have nifi version 1.9 ? If so what fix we could have for this  ?

 

 

5 REPLIES 5
Highlighted

Re: nifi 1.9 throwing error on update record while translating json to avro and the error does not explain much

Master Guru

@JasmineD 

 

"Bulletins" that you see being displayed but the processor will not include any stack traces that may also be produced.  Your bulletins all say that NiFi node "uaznifi003" is only node throwing this exception right now.
I suggest looking in the nifi-app.log on the node for this exception to see if a stack trace is being output to the log file that will help understand more about the issue.

 

Thanks,

Matt

Re: nifi 1.9 throwing error on update record while translating json to avro and the error does not explain much

New Contributor

@MattWho  thanks for your hint. Did a little more dig into bulletin which says the error is happening on other nodes too ( uaz001-uaz002 ). Processor level only showed one ( uaz003 ) though. Also the app log just outputting tons of messages , does not explicitly say anything about the particular issue. 

 

Is there anyway we can run this with debug mode on for this particular processor ?

Re: nifi 1.9 throwing error on update record while translating json to avro and the error does not explain much

New Contributor

We did some more digging at our end and this is what we have found out : 

 

1) This error is only manifesting in 1.9 version , the same code in 1.7 did not throw any error

2) this error is happening for all the nodes ( uaz001-uaz003) on nifi cluster.

3) when we re-run it with DEBUG on nifi app log it does not really show any extra info on the error front , the error reporting is as same as the one we got on the board saying  "will route to failure : null " .. 

 

Could anyone give us some more hint on this, would really appreciate that  ?

Re: nifi 1.9 throwing error on update record while translating json to avro and the error does not explain much

Master Guru

@JasmineD 

 

Unfortunately without your nifi-app.log (containing error), your processor configuration, controller services configuration and sample failing data, it is going t be very difficult to offer any suggestions as to what may be going on here.  The "null" error message is just not descriptive enough.  

Matt

Re: nifi 1.9 throwing error on update record while translating json to avro and the error does not explain much

New Contributor

Hello @MattWho  ,

That is a good point , and I understand the necessity of the details. 

here are the attachments if it helps  :

 

1) out put avro controller 

avro_writer_controller.PNGoutput avro writer

 

 

2) input json reader controllerjson_reader_controller.PNGinput json reader

 

 

3) mapping fields - json to avrojson_avro_transformation.PNGmapping between json and avro

4) sample json data 

 

 

{ 
   "ApplicationRefer":{ 
      "Id":65805,
      "BaseApplicationId":742177,
      "ReferralTriggerTypeId":1,
      "ReferralTypeId":1,
      "ReferalReasonId":0,
      "ReferalSubReasonId":0,
      "Comments":"",
      "ReferredBy":3593,
      "ReferredOn":"2019-09-27T13:14:34Z",
      "IsReviewCompletedAutomatic":false,
      "ReviewStatus":3,
      "ReviewUpdatedBy":0,
      "ReviewUpdatedOn":"2019-09-30T13:38:51Z",
      "Priority":0,
      "ReviewOutcomeStatus":0,
      "IsComplete":false
   },
   "ReferralTriggerType":{ 
      "Id":1,
      "Label":"Decision Module",
      "Order":1,
      "Is_Active":true,
      "Code":"DecisionModule"
   },
   "ReferralType":{ 
      "Id":1,
      "Label":"Auto",
      "Order":4,
      "Is_Active":true
   },
   "ReferalReasonId":{ 
      "Id":0,
      "ApplicationEnumValueCategory":0,
      "EnumValueCode":"",
      "EnumValueText":"",
      "EnumValueDescription":"",
      "IsDefault":false,
      "IsActive":false,
      "IsDeleted":false
   },
   "ReferalSubReasonId":{ 
      "Id":0,
      "ApplicationEnumValueListId":0,
      "SubEnumValueCode":"",
      "SubEnumValueText":"",
      "SubEnumValueDescription":"",
      "IsActive":false,
      "IsDeleted":false,
      "IsDefault":false
   },
   "ReferredBy":{ 
      "Name":"kkkkkk Admin",
      "Email":"kkkkk@lolz.co.uk",
      "UserName":"submitappadmin",
      "Department":"To Do"
   },
   "ReviewStatus":{ 
      "Id":3,
      "Label":"Completed Review",
      "Order":3,
      "Is_Active":true,
      "Desciption":"This status indicates if the review activity is completed - irrespective of decision.",
      "Code":""
   },
   "ReviewUpdatedBy":{ 
      "Name":"",
      "Email":"",
      "UserName":"",
      "Department":"To Do"
   },
   "CustomerCareReferralData":{ 
      "CustomerCareData":{ 
         "Id":0,
         "BaseApplicationId":0,
         "ReferralReason":0,
         "IsAdjustmentNotYetOffered":false,
         "IsAudio":false,
         "IsAuthorisedUser":false,
         "IsBraille":false,
         "IsLargePrint":false,
         "IsLongAppNeeded":false,
         "IsPaymentSchedule":false,
         "IsNoAdjustmentNecessary":false,
         "IsHoldAppUntilCustReadsTerms":false,
         "AudioTermsandConditions":false,
         "StatementOverlayYellow":false,
         "StatementOverlayBlue":false,
         "StatementOverlayPink":false,
         "StatementOverlayRed":false,
         "IsDisclosureConsentGained":0,
         "TypeOfDisclosure":0,
         "DisclosureProviderType":0,
         "DisclosureDetails":"",
         "IsReferToResolutions":0,
         "CreatedBy":0,
         "UpdatedBy":0,
         "CreatedOn":"1900-01-01T00:00:00",
         "UpdatedOn":"1900-01-01T00:00:00",
         "ApplicationManualReferId":0
      },
      "ReferralReason":{ 
         "Id":0,
         "ApplicationEnumValueCategory":0,
         "EnumValueCode":"",
         "EnumValueText":"",
         "EnumValueDescription":"",
         "IsDefault":false,
         "IsActive":false,
         "IsDeleted":false
      },
      "TypeOfDisclosure":{ 
         "Id":0,
         "ApplicationEnumValueCategory":0,
         "EnumValueCode":"",
         "EnumValueText":"",
         "EnumValueDescription":"",
         "IsDefault":false,
         "IsActive":false,
         "IsDeleted":false
      },
      "DisclosureProviderType":{ 
         "Id":0,
         "ApplicationEnumValueCategory":0,
         "EnumValueCode":"",
         "EnumValueText":"",
         "EnumValueDescription":"",
         "IsDefault":false,
         "IsActive":false,
         "IsDeleted":false
      },
      "IsDisclosureConsentGained":{ 
         "Id":0,
         "Label":"",
         "Order":0,
         "Is_Active":false
      },
      "CreatedBy":{ 
         "Name":"",
         "Email":"",
         "UserName":"",
         "Department":""
      },
      "UpdatedBy":{ 
         "Name":"",
         "Email":"",
         "UserName":"",
         "Department":""
      },
      "CustomerCareReferralReviewData":{ 
         "ReferredForResolutionReview":{ 
            "Id":0,
            "ReasonComment":"",
            "OutComeComment":"",
            "ReviewResolutionStatus":0,
            "SigmaId":"",
            "IsSSTExceptionTracker":false,
            "IsFeedbackRequired":false,
            "CATReference":"",
            "IsCallBackRequired":0,
            "ReferResolutionBy":0,
            "ReferResolutionOn":"1900-01-01T00:00:00",
            "ReviewDataUpdatedOn":"1900-01-01T00:00:00",
            "ReviewDataUpdatedBy":0
         },
         "ReviewResolutionStatus":{ 
            "Id":0,
            "ApplicationEnumValueCategory":0,
            "EnumValueCode":"",
            "EnumValueText":"",
            "EnumValueDescription":"",
            "IsDefault":false,
            "IsActive":false,
            "IsDeleted":false
         },
         "ReferResolutionBy":{ 
            "Name":"",
            "Email":"",
            "UserName":"",
            "Department":""
         },
         "ReviewDataUpdatedBy":{ 
            "Name":"",
            "Email":"",
            "UserName":"",
            "Department":""
         },
         "IsCallBackRequired":{ 
            "Id":0,
            "Label":"",
            "Order":0,
            "Is_Active":false,
            "Value":false
         },
         "ReviewResolutionSubReasonsDatas":[ 

         ]
      }
   },
   "FraudReferralReviewData":{ 
      "ApplicationFraudReferral":{ 
         "Id":65805,
         "AdditionalRemarks":"",
         "IsAutoFraudPolicyRefer":false,
         "IsAutoAMLPolicyRefer":true,
         "IsAutoPEPSAndSanctionRefer":true,
         "IsAutoPolicyRefer":false,
         "IsPepsAndSanctionOnly":true
      },
      "ApplicationFraudReview":{ 
         "Id":0,
         "IsFurtherInformationRequired":false,
         "LetterType":0,
         "LetterReRequest":0,
         "IsCIFASCase":false,
         "CIFASReason":0,
         "ManualFraudResolutionId":0,
         "FraudDeclineReason":0,
         "FraudDeclineLetter":0,
         "IsIDVerification":false,
         "IDVerificationReference":"",
         "FraudReviewedBy":0,
         "FraudReviewedOn":"1900-01-01T00:00:00",
         "FraudReviewUpdatedBy":0,
         "FraudReviewUpdatedOn":"1900-01-01T00:00:00"
      },
      "LetterType":{ 
         "Id":0,
         "ApplicationEnumValueCategory":0,
         "EnumValueCode":"",
         "EnumValueText":"",
         "EnumValueDescription":"",
         "IsDefault":false,
         "IsActive":false,
         "IsDeleted":false
      },
      "LetterReRequest":{ 
         "Id":0,
         "ApplicationEnumValueCategory":0,
         "EnumValueCode":"",
         "EnumValueText":"",
         "EnumValueDescription":"",
         "IsDefault":false,
         "IsActive":false,
         "IsDeleted":false
      },
      "CIFASReason":{ 
         "Id":0,
         "ApplicationEnumValueCategory":0,
         "EnumValueCode":"",
         "EnumValueText":"",
         "EnumValueDescription":"",
         "IsDefault":false,
         "IsActive":false,
         "IsDeleted":false
      },
      "FraudDeclineReason":{ 
         "Id":0,
         "ApplicationEnumValueCategory":0,
         "EnumValueCode":"",
         "EnumValueText":"",
         "EnumValueDescription":"",
         "IsDefault":false,
         "IsActive":false,
         "IsDeleted":false
      },
      "FraudDeclineLetter":{ 
         "Id":0,
         "ApplicationEnumValueCategory":0,
         "EnumValueCode":"",
         "EnumValueText":"",
         "EnumValueDescription":"",
         "IsDefault":false,
         "IsActive":false,
         "IsDeleted":false
      },
      "FraudReviewedBy":{ 
         "Name":"",
         "Email":"",
         "UserName":"",
         "Department":""
      },
      "FraudReviewUpdatedBy":{ 
         "Name":"",
         "Email":"",
         "UserName":"",
         "Department":""
      },
      "ManualFraudResolutionId":{ 
         "Id":0,
         "Label":"",
         "SearchLabel":"",
         "Order":0,
         "Is_Active":false
      },
      "ApplicationReferNOC":{ 
         "Id":0,
         "IsNOCFound":false,
         "IsNOCResolved":false
      },
      "ApplicationReferFraudCheckCaseyUpdate":{ 
         "Id":0,
         "IsCaseyDataUpdated":false,
         "IsPositveDecision":false
      }
   },
   "ReconReferralReviewData":{ 
      "ApplicationReconReferral":{ 
         "Id":0,
         "AdditionalRemarks":""
      },
      "ApplicationReconReview":{ 
         "Id":0,
         "IsReturnToOutbound":false,
         "PlaceOnHoldUntilDocRcvd":false,
         "ReviewedOn":"1900-01-01T00:00:00",
         "ReviewedBy":0
      },
      "FailedDPAProofDatas":[ 

      ],
      "ReviewedBy":{ 
         "Name":"",
         "Email":"",
         "UserName":"",
         "Department":""
      }
   },
   "AffordabilityReferralReviewData":{ 
      "AffordabilityReviewData":{ 
         "Id":0,
         "BaseApplicationId":0,
         "ApplicationReferId":0,
         "CheckStatus":0,
         "DocumentInvalidReasonId":0,
         "IsRequestFurtherProof":false,
         "CalculatedNetIncomeByProof":0.0,
         "LastUpdatedOn":"1900-01-01T00:00:00",
         "LastUpdateBy":0,
         "FinalReviewBy":0,
         "FinalReviewOn":"1900-01-01T00:00:00",
         "AffordaiblityReviewOutcome":0,
         "IsProofOfIncomeValnerable":false
      },
      "CheckStatus":{ 
         "Id":0,
         "Label":"",
         "Order":0,
         "Is_Active":false,
         "Value":false
      },
      "AffordabilityReviewOutcome":{ 
         "Id":0,
         "Label":"",
         "Order":0,
         "Is_Active":false
      },
      "DocumentInvalidReasonId":{ 
         "Id":0,
         "ApplicationEnumValueCategory":0,
         "EnumValueCode":"",
         "EnumValueText":"",
         "EnumValueDescription":"",
         "IsDefault":false,
         "IsActive":false,
         "IsDeleted":false
      },
      "LastUpdatedBy":{ 
         "Name":"",
         "Email":"",
         "UserName":"",
         "Department":""
      },
      "FinalReviewBy":{ 
         "Name":"",
         "Email":"",
         "UserName":"",
         "Department":""
      },
      "AffordabilityIncomeProofDatas":[ 

      ],
      "AffordabilityFailedDocumentDatas":[ 

      ],
      "AffordabilityReviewHistoryDatas":[ 

      ]
   },
   "ApplicationHDPreApprovalReferReview":{ 
      "ApplicationHDPreApprovalRefer":{ 
         "Id":0,
         "IsHDPreApprovalRefer":false,
         "IsHDPreApprovalResolved":false
      },
      "ApplicationHDPreApprovalReview":{ 
         "Id":0,
         "BaseApplicationId":0,
         "HDPreApproveReviewOutcome":0,
         "LastUpdatedOn":"1900-01-01T00:00:00",
         "LastUpdateBy":0,
         "FinalReviewBy":0,
         "FinalReviewOn":"1900-01-01T00:00:00"
      },
      "ApplicationHDPreApprovalReviewOutcome":{ 
         "Id":0,
         "Label":"",
         "Order":0,
         "Is_Active":false
      },
      "LastUpdatedBy":{ 
         "Name":"",
         "Email":"",
         "UserName":"",
         "Department":""
      }
   }
}

 

 

 

5) app nifi log screenshot nifi_app_log_error.png

6) this is what avro metadata is defined with :

 

{
  "namespace": "ApplicationReferDatas",
  "type": "record",
  "name":  "ApplicationReferHistory",
  "fields": [
   { "name": "BaseApplicationId", "type": "int"  },
   { "name": "AppReferComments", "type": "string"  },
   { "name": "AppReferId", "type": "long"  },
   { "name": "IsReviewCompletedAutomatic", "type": "boolean" },
   { "name": "ReferalReasonId", "type": "double"  },
   { "name": "ReferalSubReasonId", "type": "long"  },
   { "name": "ReferralTriggerTypeId", "type": "int"  },
   { "name": "ReferralTypeId", "type": "int"  },
   { "name": "ReferredBy", "type": "int"  },
   { "name": "ReferredOn", "type": "string"  },
   { "name": "ReviewStatus", "type": "int"  },
   { "name": "ReviewUpdatedBy", "type": "int"  },
   { "name": "ReviewUpdatedOn", "type": "string"  },
   { "name": "ReferalReasonIdValueCat", "type": "int"  },
   { "name": "ReferalReasonIdCode", "type": "string"  },
   { "name": "ReferalReasonIdDescr", "type": "string"  },
   { "name": "ReferalReasonIdText", "type": ["null", "string"] },
   { "name": "ReferalSubReasonIdValueList" , "type": ["null", "long"] },
   { "name": "ReferalSubReasonIdCode", "type": ["null", "string"]  },
   { "name": "ReferalSubReasonIdDescr", "type": "string"  },
   { "name": "ReferalSubReasonIdText", "type": ["null", "string"] },
   { "name": "ReferralTriggerTypeCode", "type": "string"  },
   { "name": "ReferralTriggerTypeLabel", "type": "string"  },
   { "name": "ReferralTypeLabel", "type": "string"  },
   { "name": "ReferredByName", "type": "string"  },
   { "name": "RefferedByUserName", "type": "string"  },
   { "name": "ReviewStatusDescr", "type": ["null", "string"] },
   { "name": "ReviewStatusLabel", "type": ["null", "string"] },
   { "name": "ReviewUpdatedByName", "type": "string"  },
   { "name": "ReviewUpdatedByUserName", "type": "string"  }
]
}

 

 

Thanks ,

Jasmine 

Don't have an account?
Coming from Hortonworks? Activate your account here