The main consequences are for running jobs some of which may depend on ats ( too late for that ) and any investigation of performance of old jobs ( which are now gone ) apart from that nothing I would know about. Would be interested to know who set the retention period to 8 years 🙂 That doesnt make any sense at all. You could have simply changed that as well he should have cleaned up the data soon as well. Hope that works. ... View more

Damn not fast enough, was about to write this, you get the column counts, types and some statistics out of it, you will have to invent the column names though. [root@sandbox ~]# hadoop fs -ls /apps/hive/warehouse/torc Found 2 items -rwxrwxrwx 3 root hdfs 16653 2016-03-14 15:35 /apps/hive/warehouse/torc/000000_0 -rwxrwxrwx 3 root hdfs 16653 2016-03-14 15:35 /apps/hive/warehouse/torc/000000_0_copy_1 [root@sandbox ~]# hive --orcfiledump /apps/hive/warehouse/torc/000000_0 WARNING: Use "yarn jar" to launch YARN applications. Processing data file /apps/hive/warehouse/torc/000000_0 [length: 16653] Structure for /apps/hive/warehouse/torc/000000_0 File Version: 0.12 with HIVE_8732 16/07/27 10:57:36 INFO orc.ReaderImpl: Reading ORC rows from /apps/hive/warehouse/torc/000000_0 with {include: null, offset: 0, length: 9223372036854775807} 16/07/27 10:57:36 INFO orc.RecordReaderFactory: Schema is not specified on read. Using file schema. Rows: 823 Compression: ZLIB Compression size: 262144 Type: struct<_col0:string,_col1:string,_col2:int,_col3:int> Stripe Statistics: Stripe 1: Column 0: count: 823 hasNull: false Column 1: count: 823 hasNull: false min: 00-0000 max: 53-7199 sum: 5761 Column 2: count: 823 hasNull: false min: Accountants and auditors max: Zoologists and wildlife biologists sum: 28550 Column 3: count: 823 hasNull: false min: 340 max: 134354250 sum: 403062800 Column 4: count: 819 hasNull: true min: 16700 max: 192780 sum: 39282210 ... View more

You still will not have HIVE_HOME because the scripts set it dynamically you need to replace that place holder with /usr/hdp/<yourversionlookitupinlinux>/hive ... View more

So on the ambari go to hosts, select the host you want and press the big Add+ button ... View more

Are you using HDP? Then you would install them through Ambari. Host-> Add Client ... View more

you need to replace hive home with the actual path? /usr/hdp/<version>/hive/ Also on the node where you run it you need a hive client installed. ... View more

@Kartik Vashishta Again you don't understand HDFS. There is no limiting factor apart from the total disc capacity of the cluster. HDFS will put blocks ( simply files on the local file system ) on the discs of your cluster and fill them up. It will also make sure that you have 3 copies of each block. There is no limit but the total size of space. Now its not very smart to have differently sized discs in your cluster because this means not all spindles will be utilized equally but there is no limit per se. The performance problem will be that the small drives will be filled up and then all write activity will happen on the bigger drives. So equal drive sizes are recommended. But again not required. So its recommended to have equally sized discs but its not a requirement. The other discs will not be empty. Also its not a requirement to have the same number of drives in each node but you need to configure each node with the correct number of drives using config groups as sagar said. ... View more

What Sagar says, Its not like RAID as in that whole discs are mirrored across nodes. Blocks are put on different nodes and HDFS will try to fill up the available space. Its pretty flexible. ... View more