thanks so much for answering , i think i am closer to answer please elaborate below solutions as advised by you , i am not much familiar with hive
Put data into Hive. There are ways to put xml data into hive. At a very dirty level you have an xpath udf to work on xml data in Hive, or you can package it luxuriously by converting xml to avro and then using serde to map the filelds to column names. (let me know if you want to go over this in more detail and I can help you there) Combine bunch of files, zip it up and upload to hdfs. This option is good, if your access is very cold (once in a while) and you are going to access the files physically (like hadoop fs -get) FYI below: Please advice on how to store and read archive data from hive while storing data in hive , should i save it as .har in hdfs ? our application generatee small size xml files which are stored on NAS and XML associated metadata in DB . plan is to extract metadata from DB into 1 file and compress xml into 1 huge archive say 10GB , suppose each archive is 10GB and data is 3 months old so i wanted to know best solution for storing and accessing this archived data in hadoop --> HDFS/HIVE/Hbase Please advise what do you think will be the better approach for reading this archived data suppose i am storing this archived data in hive so how do i retrieve this archived data Please guide me for storing archived data in hive also guide for Retrieving/reading archived data from hive when needed
... View more
