yes similar to this https://community.hortonworks.com/questions/79103/what-is-the-best-way-to-store-small-files-in-hadoo.html#comment-80387 ... View more

@mqureshiThanks a lot for your help and for guiding 🙂 . thanks for explaining in detail ... View more

thanks so much for answering , i think i am closer to answer please elaborate below solutions as advised by you , i am not much familiar with hive Put data into Hive. There are ways to put xml data into hive. At a very dirty level you have an xpath udf to work on xml data in Hive, or you can package it luxuriously by converting xml to avro and then using serde to map the filelds to column names. (let me know if you want to go over this in more detail and I can help you there) Combine bunch of files, zip it up and upload to hdfs. This option is good, if your access is very cold (once in a while) and you are going to access the files physically (like hadoop fs -get) FYI below: Please advice on how to store and read archive data from hive while storing data in hive , should i save it as .har in hdfs ? our application generatee small size xml files which are stored on NAS and XML associated metadata in DB . plan is to extract metadata from DB into 1 file and compress xml into 1 huge archive say 10GB , suppose each archive is 10GB and data is 3 months old so i wanted to know best solution for storing and accessing this archived data in hadoop --> HDFS/HIVE/Hbase Please advise what do you think will be the better approach for reading this archived data suppose i am storing this archived data in hive so how do i retrieve this archived data Please guide me for storing archived data in hive also guide for Retrieving/reading archived data from hive when needed ... View more

Hi thanks for your reply I appreciate your inputs Please advice on how to store and read archive data from hive while storing data in hive , should i save it as .har in hdfs ? our application generatee small size xml files which are stored on NAS and XML associated metadata in DB . plan is to extract metadata from DB into 1 file and compress xml into 1 huge archive say 10GB , suppose each archive is 10GB and data is 3 months old so i wanted to know best solution for storing and accessing this archived data in hadoop --> HDFS/HIVE/Hbase Please advise what do you think will be the better approach for reading this archived data suppose i am storing this archived data in hive so how do i retrieve this archived data Please guide me for storing archived data in hive also guide for Retrieving/reading archived data from hive when needed ... View more

@hduraiswamy i appreciate your inputs Please advice on how to store and read archive data from hive while storing data in hive , should i save it as .har in hdfs ? our application generatee small size xml files which are stored on NAS and XML associated metadata in DB . plan is to extract metadata from DB into 1 file and compress xml into 1 huge archive say 10GB , suppose each archive is 10GB and data is 3 months old so i wanted to know best solution for storing and accessing this archived data in hadoop --> HDFS/HIVE/Hbase Please advise what do you think will be the better approach for reading this archived data suppose i am storing this archived data in hive so how do i retrieve this archived data Please guide me for storing archived data in hive also guide for Retrieving/reading archived data from hive when needed ... View more

@mqureshiHi our application generate small size xml files which are stored on NAS and XML associated metadata in DB . plan is to extract metadata from DB into 1 file and compress xml into 1 huge archive say 10GB , suppose each archive is 10GB and data is 3 months old so i wanted to know best solution for storing and accessing this archived data in hadoop --> HDFS/HIVE/Hbase Please advise what do you think will be the better approach for reading this archived data suppose i am storing this archived data in hive so how do i retrieve this archived data Please guide me for storing archived data in hive also guide for Retrieving/reading archived data from hive when needed ... View more

Please see below options and NOTE NOTE : for both options CopyTable and Export/Import Since the cluster is up, there is a risk that edits could be missed in the export process. http://hbase.apache.org/0.94/book/ops_mgt.html#copytable CopyTable is a utility that can copy part or of all of a table, either to the same cluster or another cluster. The usage is as follows: $ bin/hbase org.apache.hadoop.hbase.mapreduce.CopyTable [--starttime=X] [--endtime=Y] [--new.name=NEW] [--peer.adr=ADR] tablename http://hbase.apache.org/0.94/book/ops_mgt.html#export 14.1.7. Export Export is a utility that will dump the contents of table to HDFS in a sequence file. Invoke via: $ bin/hbase org.apache.hadoop.hbase.mapreduce.Export <tablename> <outputdir> [<versions> [<starttime> [<endtime>]]] Note: caching for the input Scan is configured via hbase.client.scanner.caching in the job configuration. 14.1.8. Import Import is a utility that will load data that has been exported back into HBase. Invoke via: $ bin/hbase org.apache.hadoop.hbase.mapreduce.Import <tablename> <inputdir> ... View more

@da li hey please have a look at below https://community.hortonworks.com/questions/153/impersonation-error-while-trying-to-access-ambari.html if it helps accept the answer You need to create the proxy settings for 'root', since Ambari runs as root. This allows it to impersonate the user in hdfs. similar thing you need to do for oozie user , like its done for root hadoop.proxyuser.root.groups=* hadoop.proxyuser.root.hosts=* ... View more

@Scott Shaw Thanks but please see below questions 1. can i get same performance what i get in my optimized and purpose-built infrastructure HDP cluster ? because data lake is central and can i tune it specifically for 1 application ? 2. how can i manage different HDP versions in data lake ? 3. if something goes wrong with security or configuration because of 1 application then my whole data lake will be impacted ? ... View more

@Ted Yu @emaxwell @Josh Elser thanks all for your confirmation , that's why i asked if rpm is relocatable 🙂 i will close this thread But I believe it should support this from corporate IT policy POV , wherein we many times have issue putting files in /usr and /etc i should open this as an idea , WDYT ? ... View more