yes similar to this https://community.hortonworks.com/questions/79103/what-is-the-best-way-to-store-small-files-in-hadoo.html#comment-80387 ... View more

@mqureshiThanks a lot for your help and for guiding 🙂 . thanks for explaining in detail ... View more

thanks so much for answering , i think i am closer to answer please elaborate below solutions as advised by you , i am not much familiar with hive Put data into Hive. There are ways to put xml data into hive. At a very dirty level you have an xpath udf to work on xml data in Hive, or you can package it luxuriously by converting xml to avro and then using serde to map the filelds to column names. (let me know if you want to go over this in more detail and I can help you there) Combine bunch of files, zip it up and upload to hdfs. This option is good, if your access is very cold (once in a while) and you are going to access the files physically (like hadoop fs -get) FYI below: Please advice on how to store and read archive data from hive while storing data in hive , should i save it as .har in hdfs ? our application generatee small size xml files which are stored on NAS and XML associated metadata in DB . plan is to extract metadata from DB into 1 file and compress xml into 1 huge archive say 10GB , suppose each archive is 10GB and data is 3 months old so i wanted to know best solution for storing and accessing this archived data in hadoop --> HDFS/HIVE/Hbase Please advise what do you think will be the better approach for reading this archived data suppose i am storing this archived data in hive so how do i retrieve this archived data Please guide me for storing archived data in hive also guide for Retrieving/reading archived data from hive when needed ... View more

Hi thanks for your reply I appreciate your inputs Please advice on how to store and read archive data from hive while storing data in hive , should i save it as .har in hdfs ? our application generatee small size xml files which are stored on NAS and XML associated metadata in DB . plan is to extract metadata from DB into 1 file and compress xml into 1 huge archive say 10GB , suppose each archive is 10GB and data is 3 months old so i wanted to know best solution for storing and accessing this archived data in hadoop --> HDFS/HIVE/Hbase Please advise what do you think will be the better approach for reading this archived data suppose i am storing this archived data in hive so how do i retrieve this archived data Please guide me for storing archived data in hive also guide for Retrieving/reading archived data from hive when needed ... View more

@hduraiswamy i appreciate your inputs Please advice on how to store and read archive data from hive while storing data in hive , should i save it as .har in hdfs ? our application generatee small size xml files which are stored on NAS and XML associated metadata in DB . plan is to extract metadata from DB into 1 file and compress xml into 1 huge archive say 10GB , suppose each archive is 10GB and data is 3 months old so i wanted to know best solution for storing and accessing this archived data in hadoop --> HDFS/HIVE/Hbase Please advise what do you think will be the better approach for reading this archived data suppose i am storing this archived data in hive so how do i retrieve this archived data Please guide me for storing archived data in hive also guide for Retrieving/reading archived data from hive when needed ... View more

@mqureshiHi our application generate small size xml files which are stored on NAS and XML associated metadata in DB . plan is to extract metadata from DB into 1 file and compress xml into 1 huge archive say 10GB , suppose each archive is 10GB and data is 3 months old so i wanted to know best solution for storing and accessing this archived data in hadoop --> HDFS/HIVE/Hbase Please advise what do you think will be the better approach for reading this archived data suppose i am storing this archived data in hive so how do i retrieve this archived data Please guide me for storing archived data in hive also guide for Retrieving/reading archived data from hive when needed ... View more

Please see below options and NOTE NOTE : for both options CopyTable and Export/Import Since the cluster is up, there is a risk that edits could be missed in the export process. http://hbase.apache.org/0.94/book/ops_mgt.html#copytable CopyTable is a utility that can copy part or of all of a table, either to the same cluster or another cluster. The usage is as follows: $ bin/hbase org.apache.hadoop.hbase.mapreduce.CopyTable [--starttime=X] [--endtime=Y] [--new.name=NEW] [--peer.adr=ADR] tablename http://hbase.apache.org/0.94/book/ops_mgt.html#export 14.1.7. Export Export is a utility that will dump the contents of table to HDFS in a sequence file. Invoke via: $ bin/hbase org.apache.hadoop.hbase.mapreduce.Export <tablename> <outputdir> [<versions> [<starttime> [<endtime>]]] Note: caching for the input Scan is configured via hbase.client.scanner.caching in the job configuration. 14.1.8. Import Import is a utility that will load data that has been exported back into HBase. Invoke via: $ bin/hbase org.apache.hadoop.hbase.mapreduce.Import <tablename> <inputdir> ... View more

@da li hey please have a look at below https://community.hortonworks.com/questions/153/impersonation-error-while-trying-to-access-ambari.html if it helps accept the answer You need to create the proxy settings for 'root', since Ambari runs as root. This allows it to impersonate the user in hdfs. similar thing you need to do for oozie user , like its done for root hadoop.proxyuser.root.groups=* hadoop.proxyuser.root.hosts=* ... View more

@Scott Shaw Thanks but please see below questions 1. can i get same performance what i get in my optimized and purpose-built infrastructure HDP cluster ? because data lake is central and can i tune it specifically for 1 application ? 2. how can i manage different HDP versions in data lake ? 3. if something goes wrong with security or configuration because of 1 application then my whole data lake will be impacted ? ... View more

@Ted Yu @emaxwell @Josh Elser thanks all for your confirmation , that's why i asked if rpm is relocatable 🙂 i will close this thread But I believe it should support this from corporate IT policy POV , wherein we many times have issue putting files in /usr and /etc i should open this as an idea , WDYT ? ... View more

Thanks a lot Kuldeep i agree and thats why i wanted suggestions from experts like you 🙂 ... View more

We can store application related data and logs on SAN/NAS However SAN/NAS are not at all recommended for I/O sensitive and CPU bound jobs , that is to avoid bottleneck situations while reading data from disk or from network or in processing data So for Logs/application data --> SAN/NAS Data nodes data --> DAS with JBOD configuration NO RAID NN/SN/JT nodes --> should be highly available [ RAID 5/10(depends on usecase) ] Hadoop is a scale out and shared nothing architecture http://www.bluedata.com/blog/2015/12/separating-hadoop-compute-and-storage/ https://community.emc.com/servlet/JiveServlet/previewBody/41473-102-1-132603/Virtualizing%20Hadoop%20in%20Large%20Scale%20Infrastructures.pdf Also I understand sometimes true cost of DAS is also more considering Hadoop replication , but this is how Hadoop is thriving (One of the key tenets of Hadoop is to bring the compute to the storage instead of the storage to the compute.) ... View more

@Sbandaru: i researched over this more deeply and conclusion is , we don't need edge node We don’t need edge node if Hadoop cluster and application are in same network its only needed when hadoop cluster and application are in diff network , at that time edge node acts as a gateway to hadoop cluster ( like a proxy ) thanks for your inputs ... View more

will be helpful ... View more

thanks nice ... View more

Thanks a lot ... View more

answer looks goo d.. Thanks for your answer can you please advise how to decide DiskIO in cluster ? which factors to consider for Disk I/O calculation ? ... View more