Hello, I'm still new to Hadoop technology and I'm struggling defining the best approach for the following 2 similar challenges: In first instance, trying to ingest the files -> Hive. InferAVROSchema in Nifi is limited as it does not always recognize the right data type, generating a fair amount of error when the files are ingested. Switching to specifiy the schema manually bring the following problems: - Ingesting CSV files that have schema updates over the year, I have a versioning documentation giving me the schema changes, however the date in the versioning document do not match the date of effective change. - Ingesting hourly CSV files with a schema depending of the business activity (a set of columns is mandatory, a large set is optionnal and will only be seen when the underling options have been used) . The schema of the files is different from hours to hours, and I can't predict which one is to expect. My feelings are that I have to move to NoSql type of DB / storage, but I'm not exactly sure how to tackle this in the best way. Has anyone faced similar problematic? Thanks Christophe
... View more
