Member since
01-25-2018
6
Posts
1
Kudos Received
0
Solutions
01-27-2018
08:28 PM
@Shu Thanks. This worked. Do you know if i can automate this flow using this external output file in avro format to load into a hive table?
... View more
01-26-2018
04:32 PM
@Shu Attaching the source file i have. Below are the Avroschemaregistry structure: {
"namespace": "nifi",
"name": "balances",
"type": "record",
"fields": [
{ "name": "Source_Code", "type": "string" },
{ "name": "Source_Product_ID", "type": "int" },
{ "name": "Source_Party_ID", "type": "int" },
{ "name": "Govt_Issued_Tax_ID", "type": "int" },
{ "name": "Prefix_Name", "type": "string" },
{ "name": "First_Name", "type": "string" },
{ "name": "Middle_Name", "type": "string" },
{ "name": "Last_Name", "type": "string" },
{ "name": "Suffix_Name", "type": "string" },
{ "name": "Full_Legal_Name", "type": "string" },
{ "name": "NickName", "type": "string" },
{ "name": "Birth_Date", "type": "int", "logicalType":"date" },
{ "name": "Birth_Place_Name", "type": "string" },
{ "name": "Gender", "type": "string" },
{ "name": "Smoker_Indicator", "type": "string" },
{ "name": "Risk_Class", "type": "string" },
{ "name": "Marriage_Status_Code", "type": "string" },
{ "name": "Agreement_Name", "type": "string" },
{ "name": "Status_Code", "type": "string" },
{ "name": "Payment_Method", "type": "string" },
{ "name": "Agreement_Original_Inception_Date", "type": "int", "logicalType":"date" },
{ "name": "Payment_Frequency", "type": "string" },
{ "name": "Payment_Frequency_Desc", "type": "string" },
{ "name": "Agreement_Covered_Amount", "type": "int" },
{ "name": "Agreement_Number", "type": "string" },
{ "name": "Agreement_Party_Role", "type": "string" },
{ "name": "Effective_Date", "type": "int", "logicalType":"date" },
{ "name": "Expiration_Date", "type": "int", "logicalType":"date" }
]
} Below are the schema from CSVRecordSerWriter {
"namespace": "nifi",
"name": "balances",
"type": "record",
"fields": [
{ "name": "Source_Code", "type": "string" },
{ "name": "Source_Product_ID", "type": "int" },
{ "name": "Source_Party_ID", "type": "int" },
{ "name": "Govt_Issued_Tax_ID", "type": "int" },
{ "name": "Prefix_Name", "type": "string" },
{ "name": "First_Name", "type": "string" },
{ "name": "Middle_Name", "type": "string" },
{ "name": "Last_Name", "type": "string" },
{ "name": "Suffix_Name", "type": "string" },
{ "name": "Full_Legal_Name", "type": "string" },
{ "name": "NickName", "type": "string" },
{ "name": "Birth_Date", "type": "int", "logicalType":"date" }
]
}
I have given the date format as MM/dd/yy. I am attaching the error too. Is there something which i am doing wrong? P.S: I changed the data in govt_issue_tax_id to make it integer and still the same error. I changed the date format as per your format and still error with date
... View more
01-26-2018
01:49 AM
@Shu Thanks for the reply. My source file have below column: Source Code,Source Product ID,Source Party ID,Govt Issued Tax ID,Prefix Name,First Name,Middle Name,Last Name,Suffix Name,Full Legal Name,NickName,Birth Date,Birth Place Name,Gender,Smoker Indicator,Risk Class,Marriage Status Code,Agreement Name,Status Code,Payment Method,Agreement_Original_Inception_Date,Payment_Frequency,Payment_Frequency_Desc,Agreement_Covered_Amount,Agreement_Number,Agreement_Party_Role,Effective_Date,Expiration_Date,Agreement_Premium_Amount,Line_Of_Business_Identifier,Licensed_Product_Name,Product_Description,Product_Code,Address_Line1,Address_Line2,Region,Location_Code,Location_Name(City),Location_Number,State(State),State_code,Country,Coverage_Part_Code,Coverage_Type_identifier,Coverage_name,Coverage_Description,Coverage_Group_Identifier,Coverage_Code,Coverage_Term_Period,Coverage_Gender,Coverage_Class,Agreement_Type_Code,Party_Type_Code,County_Name,Coverage_Type_Code I think i am having issue because of data type in avro. My goal is to ultimately load all these files into a hive table. Can you please help here specially with date type?
... View more
01-26-2018
01:03 AM
@Shu Hi, It didnt work on my csv files as its very large with almost 40 field. Having issue in converting to avro. I see in your xml, avro schema has been hard coded. Is it possible to auto generate from the csv file i have ?
... View more
01-25-2018
06:12 PM
1 Kudo
Hello, I have a csv files with multiple attribute with header name: Source FIle: Name, Age, Sex, Country, City, Postal Code I want to break this csv on the basis of attribute name in 3 separate csv file: File1: Name, Age, Country File2: Name, Country, City File 3: Country,. City, Postal code Can someone please help me how can i do this into nifi?
... View more
Labels:
- Labels:
-
Apache NiFi
01-25-2018
03:25 AM
how can you set transactional'='true' for external table in hive? I dont think its possible.
... View more