I'm trying to ingest csv files with variable (and unpredictable) schéma to Hbase.
Based on @Binu Mathew advice, I started to use spark with Databricks. So far I can infer the schema and read the file into a data frame.
I'm trying to write the data frame to a Hbase table, while using the inferred schéma as catalog. But I could not find any way to transform the schéma (df.dtypes) into a valid catalog.
Has anyone any advices or link to advice on how to proceed ?