@Shu Thanks. This worked. Do you know if i can automate this flow using this external output file in avro format to load into a hive table? ... View more

@Shu Attaching the source file i have. Below are the Avroschemaregistry structure: { "namespace": "nifi", "name": "balances", "type": "record", "fields": [ { "name": "Source_Code", "type": "string" }, { "name": "Source_Product_ID", "type": "int" }, { "name": "Source_Party_ID", "type": "int" }, { "name": "Govt_Issued_Tax_ID", "type": "int" }, { "name": "Prefix_Name", "type": "string" }, { "name": "First_Name", "type": "string" }, { "name": "Middle_Name", "type": "string" }, { "name": "Last_Name", "type": "string" }, { "name": "Suffix_Name", "type": "string" }, { "name": "Full_Legal_Name", "type": "string" }, { "name": "NickName", "type": "string" }, { "name": "Birth_Date", "type": "int", "logicalType":"date" }, { "name": "Birth_Place_Name", "type": "string" }, { "name": "Gender", "type": "string" }, { "name": "Smoker_Indicator", "type": "string" }, { "name": "Risk_Class", "type": "string" }, { "name": "Marriage_Status_Code", "type": "string" }, { "name": "Agreement_Name", "type": "string" }, { "name": "Status_Code", "type": "string" }, { "name": "Payment_Method", "type": "string" }, { "name": "Agreement_Original_Inception_Date", "type": "int", "logicalType":"date" }, { "name": "Payment_Frequency", "type": "string" }, { "name": "Payment_Frequency_Desc", "type": "string" }, { "name": "Agreement_Covered_Amount", "type": "int" }, { "name": "Agreement_Number", "type": "string" }, { "name": "Agreement_Party_Role", "type": "string" }, { "name": "Effective_Date", "type": "int", "logicalType":"date" }, { "name": "Expiration_Date", "type": "int", "logicalType":"date" } ] } Below are the schema from CSVRecordSerWriter { "namespace": "nifi", "name": "balances", "type": "record", "fields": [ { "name": "Source_Code", "type": "string" }, { "name": "Source_Product_ID", "type": "int" }, { "name": "Source_Party_ID", "type": "int" }, { "name": "Govt_Issued_Tax_ID", "type": "int" }, { "name": "Prefix_Name", "type": "string" }, { "name": "First_Name", "type": "string" }, { "name": "Middle_Name", "type": "string" }, { "name": "Last_Name", "type": "string" }, { "name": "Suffix_Name", "type": "string" }, { "name": "Full_Legal_Name", "type": "string" }, { "name": "NickName", "type": "string" }, { "name": "Birth_Date", "type": "int", "logicalType":"date" } ] } I have given the date format as MM/dd/yy. I am attaching the error too. Is there something which i am doing wrong? P.S: I changed the data in govt_issue_tax_id to make it integer and still the same error. I changed the date format as per your format and still error with date ... View more

@Shu Thanks for the reply. My source file have below column: Source Code,Source Product ID,Source Party ID,Govt Issued Tax ID,Prefix Name,First Name,Middle Name,Last Name,Suffix Name,Full Legal Name,NickName,Birth Date,Birth Place Name,Gender,Smoker Indicator,Risk Class,Marriage Status Code,Agreement Name,Status Code,Payment Method,Agreement_Original_Inception_Date,Payment_Frequency,Payment_Frequency_Desc,Agreement_Covered_Amount,Agreement_Number,Agreement_Party_Role,Effective_Date,Expiration_Date,Agreement_Premium_Amount,Line_Of_Business_Identifier,Licensed_Product_Name,Product_Description,Product_Code,Address_Line1,Address_Line2,Region,Location_Code,Location_Name(City),Location_Number,State(State),State_code,Country,Coverage_Part_Code,Coverage_Type_identifier,Coverage_name,Coverage_Description,Coverage_Group_Identifier,Coverage_Code,Coverage_Term_Period,Coverage_Gender,Coverage_Class,Agreement_Type_Code,Party_Type_Code,County_Name,Coverage_Type_Code I think i am having issue because of data type in avro. My goal is to ultimately load all these files into a hive table. Can you please help here specially with date type? ... View more

@Shu Hi, It didnt work on my csv files as its very large with almost 40 field. Having issue in converting to avro. I see in your xml, avro schema has been hard coded. Is it possible to auto generate from the csv file i have ? ... View more

how can you set transactional'='true' for external table in hive? I dont think its possible. ... View more