For more information on the record processing philosophy and some examples, see Record-oriented data with NiFi and Apache NiFi Records and Schema Registries.
- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Mapping different header names to hive column names in nifi flow
- Labels:
-
Apache Hive
-
Apache NiFi
-
NiFi Registry
Created on
‎08-27-2019
08:13 PM
- last edited on
‎08-28-2019
06:58 AM
by
cjervis
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
I am trying to see how to map different fields names (for same data) due to data is from various sources, but ingest to same hive table.
Ex:
Table1 with name "Users", Columns {"firstName", "lastName", "street","city","zip","country"}
Table2 with name "UserDetails", Columns {"fName","lName","street","city","pincode","country","contact"}
Table3 with name "User", Columns {"firstName","lastname","address1","address2", "city","state", "pin", "country"}
Target Hive Table: "Users" with columns {"firstName","lastName", "address1", "address2", "city", "state", "zip", "country", "contact"}
firstName, fName maps to firstName
lastName, lName maps to lastName
street and address1 maps to address1
address2 maps to address2
zip, pincode maps to zip
...so on.
What is the best way to map these different column headers to same hive columns in nifi? Appreciate any thoughts. If there is a way I can create only one flow template to handle these that would be great as well.
Appreciate your help.
Thanks,
Ravi
Created ‎08-28-2019 11:15 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
As explained elsewhere by Andy:
You can accomplish this with a ConvertRecord processor. Register an Avro schema describing the expected format in a Schema Registry (controller service), and create a CSVReader implementation to convert this incoming data to the generic Apache NiFi internal record format. Similarly, use a CSVRecordSetWriter with your output schema to write the data back to CSV in whatever columnar order you like.
- Dennis Jaheruddin
If this answer helped, please mark it as 'solved' and/or if it is valuable for future readers please apply 'kudos'.
