Hello everyone , I have to update and maybe delete some rows in existing hive tables ( orc ) , unfortunately the update and delete operations are blocked by the datalake management team . So , i would like to know if there is any way to do the update without direct acid operations thx ... View more

Hello , I want to load data from csv file from different Datasources in HDFS to ORC table , including some data validations ( business rules ...) and transformation ... . Actually , my general process is to load all csv files with the same structure from the same datasource to one unique external table , then processing and applying the validation rules and the data transformation , then load the data to orc table . My question is , what is the best way to automate the process (loading , validation and transformation ) to make scheduling and monitoring easy and also all the CRUD problems . PS : i started working with java spark and oozie for scheduling thx ... View more