Hi Lars,
Yes I have read I believe most of the articles and the doco writen on Kite SDK.
However, my partition fields (year, month, day) are not part of my data files, and there is no date or timestamp field that tells me that this data is of today or a month ago.
My partition config (if I can use one) would be:
[{
"type" : "provided",
"name" : "year"
},
{
"type" : "provided",
"name" : "month"
},
{
"type" : "provided",
"name" : "day"
}]
And when I want to csv-import or json-import my files I don't see how to tell kitesdk-dataset explicitly that I want to store the imported file in partition year=2016, month=05, day=30.
Right now this is what I am doing: I create a dataset, create a partition directory and then copy the parquet file to it):
> kite-dataset csv-schema ods_ml_au.Introducer_Group_30_05_2016.psv --class IntroducerGroup --delimiter '|' -o introducerGroup.avsc
> hdfs dfs -put introducerGroup.avsc /user/caf/macleasing/format
> kite-dataset create dataset:hdfs:/user/caf/macleasing/stage/ml/introducerGroups -s hdfs:/user/caf/macleasing/format/introducerGroup.avsc -f parquet
> hdfs dfs -put ods_ml_au.Introducer_Group_30_05_2016.psv /user/caf/macleasing/source
> kite-dataset csv-import hdfs:/user/caf/macleasing/source/ods_ml_au.Introducer_Group_30_05_2016.psv dataset:hdfs:/user/caf/macleasing/stage/ml/introducerGroups --delimiter '|'
> hdfs dfs -mkdir -p /user/caf/macleasing/stage/ml/introducerGroups/year=2016/month=05/day=30
>hdfs dfs -mv /user/caf/macleasing/stage/ml/introducerGroups/*.parquet /user/caf/macleasing/stage/ml/introducerGroups/year=2016/month=05/day=30/
How can I avoid the explicit creation of directory and file movement??
I want to use my partition-config
Cheers Khalef
